Tvtv.us site.ini doesn't work anymore

the ini file is the same as what I posted a few messages before. I will add a smaller version of my config here as my current config has 900+ channels, feel free to direct message me and I will help you work it out.

Hi all,

I actually made the tvtv.us.ini on the siteini repo originally. I'm hopefully going to update that version, which will work its way into the standard release. But, I wonder if they'll step up their game once they see we've bypassed this minor attempt to stop us (removing stationid from programme list requests and the robots.txt business).

With regard to the robots.txt issue. Is it the case that WG+ should do what it's doing? I had to attach wireshark to see what was actually happening. From what I can see, when the robots.txt had that text in there, WG+ was overriding whatever you had set in the config for user-agent, and was always sending a user-agent of "WebGrab+Plus". With that user agent the tvtv.us site will return 403. Once you remove the robots.txt, WG+ sends requests with the correct user-agent set from the config and thus you get the results you expect.

Seems weird that the robots.txt makes WG+ do this. From what I can see, the standard doesn't suggest this should be the case at all. As it is, this seems clear they don't want us scraping their data though. So I don't think this will be the last counter-measure I expect.

The tVTV.us site as stopped allowing grabbing again, Was working fine last thursday 13/02/2020, but now just getting the warning, This repeats for all the 95 channels just a few here from logs, any info would be great.

[ Info ] ( 1/95 ) TVTV.US -- chan. (xmltv_id=AMC - Eastern Feed) -- mode Force
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Error ] Unable to update channel AMC - Eastern Feed
[Critical] Generic syntax exception:
[Critical] message:
[Error ] no index page data received from AMC - Eastern Feed
[Error ] unable to update channel, try again later
[ Info ] Existing guide data restored!
[ Debug ]
[ Info ] ( 1/95 ) TVTV.US -- chan. (xmltv_id=American Heroes Channel) -- mode Force
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Error ] Unable to update channel American Heroes Channel
[Critical] Generic syntax exception:
[Critical] message:
[Error ] no index page data received from American Heroes Channel
[Error ] unable to update channel, try again later
[ Info ] Existing guide data restored!

leadersat wrote:

The tVTV.us site as stopped allowing grabbing again, Was working fine last thursday 13/02/2020, but now just getting the warning, This repeats for all the 95 channels just a few here from logs, any info would be great.

Yeah, it happens from time to time that the robots file gets updated. Look at the robots/tvtv.us.robots file. If all the stuff is in there again delete it (not the file, the stuff in the file and save).

On linux I just changed the mode to make it read only and it hasn't happened since. Might be possible for you?
[/quote]

Follow instructions of site.ini (see siteini.pack)

Thank you rooty, that as sorted it a treat. working perfect again.

again it don't work annymore :(

all good for me,no changes...

Group (0) :
update requested for - 1 - out of - 1 - channels for 1 day(s)
( 1/1 ) TVTV.US -- chan. (xmltv_id=CBS (WCBS) New York, NY) -- mode Force
innnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
0.48 sec/update
Summary for update of CBS (WCBS) New York, NY
missing shows added 0
changed shows updated 0
new shows added 32
unchanged shows inspected 0
total after update 32

elapstime / updated show 0.49 seconds

Job finished at 02/04/2020 10:18:08 done in 21s

strange, here nothing changed and get reaction that the channels not anymore can be grabbed, have also upgrade the software of webgrab plus to version 3.0 but no result,
it's 2 days that this don't work anymore :(

I am usin V3.0.0

all I can say is check ur /robots/tvtvus.txt file(whever ur wg config is) and male sure the dsiisallllow li n removed.
then make it read only.

ok on my windows pc I can not do this but on my linux server I can try this to let it work that the robots maps is read only,
how can I do this on windows 10 pc also to test this ????
strange that this works for a month and now since 2 days it will not work anymore :(

i use tvtv New York

and this i get in the webgrab log file:

update requested for - 1 - out of - 1 - channels for 1 day(s)
[ Debug ]
[ Info ] ( 1/1 ) TVTV.US -- chan. (xmltv_id=tvtv New York) -- mode Incremental
[Warning ]
[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!
[Warning ] it is advised to disable this channel / site from your channel list
[Warning ]
[Error ] Unable to update channel tvtv New York
[Critical] Generic syntax exception:
[Critical] message:
[Error ] no index page data received from tvtv New York
[Error ] unable to update channel, try again later
[ Info ] Existing guide data restored!
[ Debug ]
[ Debug ] 0 shows in 1 channels
[ Debug ] 0 updated shows
[ Debug ] 0 new shows added

masterke wrote:

ok on my windows pc I can not do this but on my linux server I can try this to let it work that the robots maps is read only,
how can I do this on windows 10 pc also to test this ????
strange that this works for a month and now since 2 days it will not work anymore :(

In windows right click >properties>general> mark read-only (after modification)

ok check, did it and now it works again on pc windows,
thx, that I don't think about that, so simple solution :p

OK - I'm really sorry - I must be being stupid here - forgive me.

I have the latest INI and XML (v2.1)
I have the robots file (tvtv.us.robots) with the two lines in. (Attached)

If I make the robots file read only I get this message:
[Critical]
Access to the path 'C:\Users\Martin Conroy\AppData\Local\WebGrab+Plus\robots\tvtv.us.robots' is denied.
[Critical]
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalDelete(String path, Boolean checkHost)
at  .․. (String ,   )
at  .―. (String[] )
[Critical] Unhandled Exception
[Critical] For detailed info, see log file C:\Users\Martin Conroy\AppData\Local\WebGrab+Plus\WebGrab++.log.txt
[Critical] Execution stopped

Without it I get the grabbing not allowed message.

Sorry if I'm doing something obviously wrong, but help would be appreciated.

Sorry if I'm being really stupid but I just can't get this working!

I have the latest INI and XML (v2.1)
I have the robot file as indicated in place (attached)
I'm adding a single channel to my config to test:
NBA TV USA

If I make the robots file read-only I get the below error, and without read-only it says it can't grab an index file.

Any help would be appreciated :)

[Critical]
Access to the path 'C:\Users\Martin Conroy\AppData\Local\WebGrab+Plus\robots\tvtv.us.robots' is denied.
[Critical]
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalDelete(String path, Boolean checkHost)
at  .․. (String ,   )
at  .―. (String[] )
[Critical] Unhandled Exception
[Critical] For detailed info, see log file C:\Users\Martin Conroy\AppData\Local\WebGrab+Plus\WebGrab++.log.txt
[Critical] Execution stopped

martinconroy wrote:

Sorry if I'm being really stupid but I just can't get this working!
I have the latest INI and XML (v2.1)
I have the robot file as indicated in place (attached)
I'm adding a single channel to my config to test:
NBA TV USA
If I make the robots file read-only I get the below error, and without read-only it says it can't grab an index file.
Any help would be appreciated :)

You're right. I had the same problem happen the other night (I made the file immutable so it's impossible for the user WG+ runs as to undo). You can't do it this way. There's no graceful failure in WG+ for not being able to write to robots.txt.

Either you update it whenever you see the error happen and re-run, or you can try my current solution, which is to use the touch unix tool to make sure the date of the modified robots file is always recent. Just working on the assumption the file data is used to decide when to update from the website version. We'll have to wait and see until the next "update" to see if it works for sure.

Thanks r00ty. Do you or anyone else have any recommendations for good alternatives for US sport listings?

martinconroy wrote:

Thanks r00ty. Do you or anyone else have any recommendations for good alternatives for US sport listings?

Not so much. I mostly collect listings for UK channels. But since I have US channels I build an EPG for those too.

In general I use Schedules direct (subscription based, and not using WG+ to fetch) and WG+ with tvtv.us for US channels. Most of the nationwide channels I get on schedules direct, mostly local and missing stuff via tvtv.us. Couldn't tell you right now if most of the sports channels are there though.

Just wanted to say thanks for pointing me to Schedules Direct. An awesome tool and well worth the money for a stable reliable data source! Exactly what I needed. Thanks @r00ty

martinconroy wrote:

Just wanted to say thanks for pointing me to Schedules Direct. An awesome tool and well worth the money for a stable reliable data source! Exactly what I needed. Thanks @r00ty

That's great. Just be aware that I think WG+ only supports the older schedules direct API (which will still be fine for US listings I think).

If you want to use (or try) with their JSON API you can try this tool I made. I just now uploaded the latest changes I made to fix some performance issues. Source code is there if you want to build from source.

https://github.com/M0OPK/SDJSharp/releases (mod if this is against the rules please remove, but this doesn't compete with WG+, it purely supports the Schedules direct JSON feature WG+ doesn't have).

How I do it, is use the GUI version to select schedules direct lineups and set channel renames to match my provider, save the config. Then use the CLI version (loading the config the UI version saved) as part of a linux script.

That script first runs WG+ to pull down all latest programs for linups that are supported there. Then, I run the SDSharp to pull down Schedules direct lineups, then run SDSharp again with --importxmltv to import the WG+ xmltv it make a combined version. Which I then move to the webserver folder and it can be loaded by Kodi/imported into whatever end tool you use the xmltv file with.

I wanted to direct mail but I can't see how to do that here. Probably it's disabled.

Thanks - the API is working fine for my current needs, but I've noted the details of your tool should I need it later. Thanks very much!

I just installed version 3.1 on Windows 10 + new tvtv.us.ini, file tvtv.us.robots is set "read only" with 2 lines content: User-agent: *
User-agent: WebGrab+Plus

Every thing seem corrected but I still got this error:

[ ]
[ ] WebGrab+Plus/w MDB & REX Postprocess -- version V3.1.0.0
[ ]
[ ] Jan van Straaten
[ ] Francis De Paemeleere
[ ]
[ ] thanks to Paul Weterings and all the contributing users
[ ] --------------------------------------------------------------------------------
[ ]
[ ] Job started at 16/06/2020 13:42:17
[ Debug ]
[ Debug ] Running on: Microsoft Windows NT 6.2.9200.0
[ Debug ] Environment: 4.0.30319.42000
[ Debug ]
[ Debug ] Loading timezone data
[ Debug ] Embedded timezones source: timezone.timezonesdata.txt
[ Debug ] Reading config file: C:\Users\Phillip\AppData\Local\WebGrab+Plus\WebGrab++.config.xml
[ Info ] Checking License ..
[ Info ] For License request/update data, see WGLicense.log.txt
[ Debug ]
[ Info ] found: C:\Users\Phillip\AppData\Local\WebGrab+Plus\siteini.user\tvtv.us.ini -- Revision 05
[ Debug ] xmltv input file - P:\Public Folder\xml\sports\xmltv13.xml - found
[ Debug ] 0 superfluous shows removed
[ Info ] Found existing channel (xmltv_id=Pac-12 Bay Area) in the config file
[ Info ] Found existing channel (xmltv_id=Pac-12 LA) in the config file
[ Info ] Found existing channel (xmltv_id=Pac-12 Oregon) in the config file
[ Info ] Found existing channel (xmltv_id=Pac-12 Washington) in the config file
[ Info ] Found existing channel (xmltv_id=Pac-12 Arizona) in the config file
[ Info ] Found existing channel (xmltv_id=Pac-12 Mountain) in the config file
[ Info ]
[ Info ]
[ Info ] i=index .=same c=change g=gab r=replace n=new
[ Info ]
[ Info ]
[ Info ] Group (0) :
[ Info ] update requested for - 6 - out of - 6 - channels for 3 day(s)
[ Debug ]
[ Info ] ( 1/6 ) TVTV.US -- chan. (xmltv_id=Pac-12 Bay Area) -- mode Incremental
[ ] Job finished at 16/06/2020 13:42:18 done in 0s
[Critical]
Access to the path 'C:\Users\Phillip\AppData\Local\WebGrab+Plus\robots\tvtv.us.robots' is denied.
[Critical]
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalDelete(String path, Boolean checkHost)
at  .․. (String ,   )
at  .―. (String[] )
[Critical] Unhandled Exception
[Critical] For detailed info, see log file C:\Users\Phillip\AppData\Local\WebGrab+Plus\WebGrab++.log.txt
[Critical] Execution stopped

What's wrong here? Please help me! Thanks a lot !

philng wrote:

I just installed version 3.1 on Windows 10 + new tvtv.us.ini, file tvtv.us.robots is set "read only" with 2 lines content: User-agent: *
User-agent: WebGrab+Plus
Every thing seem corrected but I still got this error:

In this comment: http://www.webgrabplus.com/comment/20622#comment-20622 I recognized this is a problem with the read-only plan. Currently for me, as I run on linux I've just added "touch " into the bash script. For windows, not sure there's an equivalent. But if you have the info you need in the file and make sure you update the date/time on the file, at least so far for me it's working.

r00ty wrote:

philng wrote:
I just installed version 3.1 on Windows 10 + new tvtv.us.ini, file tvtv.us.robots is set "read only" with 2 lines content: User-agent: *
User-agent: WebGrab+Plus
Every thing seem corrected but I still got this error:

In this comment: http://www.webgrabplus.com/comment/20622#comment-20622 I recognized this is a problem with the read-only plan. Currently for me, as I run on linux I've just added "touch " into the bash script. For windows, not sure there's an equivalent. But if you have the info you need in the file and make sure you update the date/time on the file, at least so far for me it's working.

Thank you for your instruction, but I am unfamiliar with Linux. Now just looking for a solution for Windows.
I don't understand "But if you have the info you need in the file and make sure you update the date/time on the file",
could you explain it?

philng wrote:

Thank you for your instruction, but I am unfamiliar with Linux. Now just looking for a solution for Windows.
I don't understand "But if you have the info you need in the file and make sure you update the date/time on the file",
could you explain it?

So, the "touch" command effectively just updates the date on a file to the current date/time. So you can simulate this by opening the robots file in notepad and then saving right away. I use touch because I have my update run on a cron job automatically.

So you need to make them read/write again and then make sure the file is saved/updated just before running WG+. That's my current theory to get round the issue. Not to say it definitely will. But I've not had a problem for quite some time now since changing to that method.

If you use linux make sure you have mono > 5.0
For both (linux and windows) under webgrab folder there is a folder "robots" you will find a file named tvtv.us.robots
1. Edit so to keep only the 2 line as per attached picture and save
2. Right click on the file tvtv.us.robots > properties and change attributes to read only. For linux it will be on command line chmod 0444 tvtv.us.robots

mat8861 wrote:

If you use linux make sure you have mono > 5.0
For both (linux and windows) under webgrab folder there is a folder "robots" you will find a file named tvtv.us.robots
1. Edit so to keep only the 2 line as per attached picture and save
2. Right click on the file tvtv.us.robots > properties and change attributes to read only. For linux it will be on command line chmod 0444 tvtv.us.robots

On Windows I did exactly what you show but when run Webgrabplus i got the error:
Access to the path 'C:\Users\Phillip\AppData\Local\WebGrab+Plus\robots\tvtv.us.robots' is denied.

Always in properties, go to security tab and make sure your user "Phillip" has full rights

Yes, 'Phillip has full right' and file tvtv.us.robots has been read only.
the error
"[Critical]
Access to the path 'C:\Users\Phillip\AppData\Local\WebGrab+Plus\robots\tvtv.us.robots' is denied.
[Critical]" when I run Webgrab.

If it not be set "read only" I got the error "[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!"

All errors extract from webgrabplus_log.txt

philng wrote:

If it not be set "read only" I got the error "[Warning ] !! -- WARNING : tvtv.us doesn't allow epg grabbing !!"

Yep, like I said before. Read only as an idea didn't turn out to work. WG+ will stop on the exception when it cannot write to the file.

The best solution is to load the file, edit it to make sure it's OK (I just blank mine) and save it. Then run again it will be fine.

If you're able to refresh the date on the file (load into notepad/save again) before running WG every time that should I think prevent the problems in the future.

That's what the touch command on linux does.

ik have the same problem here, it's like with the new update of windows 10 that problem is started, verry strange.

I just talk about Windows because I am not familiar with Linux and I don't have a linux computer neither.

So we need a script like *.bat because we can't do this fast enough manually

1. Open file tvtv.us.robots to check it only has 2 lines User-agent: * User-agent: WebGrab+Plus
2. Close tvtv.us.robots then set it read only (at this time it get a new date/time)
3. Run Webgrabplus.exe

If this script working, why not Webgrabplus.exe adds it (coding) before run?

don't know what you guys are doing, my robots folder is read only and the same for tvtv.robots. WG++ only needs to read that file not to update it or write in anyway.It works since more then a year with same file on both linux and windows 10 (licensed with all the updates).

yes , you are correct. No need to write to robots file, but why we have to set it read only? if not it will be written 3 more lines:
Disallow: /tvm/
Disallow: /gn/
Disallow: /

Or maybe your computer has something supports WG++ that is not on our computer