You are here

Schedules direct channels.xml generation

13 posts / 0 new
Last post
videogamingtown
Offline
Joined: 9 years
Last seen: 6 years
Schedules direct channels.xml generation

I'm trying to generate my lineup into a channels xml file to use with schedules direct, i've followed all instructions on the site and the info document but i still cannot generate a channels.xml file

i get the following error when i run webgrab with the dummy line added to the configuration file


[ Info ] Group (0) :
[ Info ] update requested for - 1 - out of - 1 - channels for 1 day(s)
[ Debug ]
[ Info ] ( 1/1 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=dummy) -- mode Force
[Error ] Unable to update channel dummy
[Critical] Generic syntax exception:
[Critical] message:
[Error ] no index page data received from dummy
[Error ] unable to update channel, try again later
[ Info ] Existing guide data restored!
[ Debug ]
[ Debug ] 0 shows in 1 channels
[ Debug ] 0 updated shows
[ Debug ] 0 new shows added

below is my code for the ini file (i've replaced my user and pass)


**------------------------------------------------------------------------------------------------
* @header_start
* WebGrab+Plus ini for grabbing EPG data from TvGuide websites
* @Site: schedulesdirect.org
* @MinSWversion: V1.1.1/56.6
* @Revision 3 - [15/11/2015] Jan van Straaten
* - remove of faulty credits role solution
* @Revision 2 - [24/10/2015] Francis De Paemeleere
* - speedup parsing (tested with V1.56.6)
* @Revision 1 - [23/10/2014] Jan van Straaten
* - adapted site changes
* @Revision 0 - [31/08/2013] Jan van Straaten / Francis De Paemeleere
* - creation
* @Remarks: You need a login and password for this site
* @header_end
**------------------------------------------------------------------------------------------------

site {url=schedulesdirect.org|timezone=UTC|maxdays=16.1|cultureinfo=en-US|charset=UTF-8|titlematchfactor=90|keepindexpage|firstshow=1}
site {ratingsystem=MPAA|subtitlestype=teletext|episodesystem=onscreen}
url_index {url|http://dd.schedulesdirect.org/schedulesdirect/tvlistings/xtvdService}
url_index.headers {method=SOAP}
url_index.headers {customheader=SOAPAction=urn:tvDataDelivery#download}
url_index.headers {customheader=Accept-Encoding=gzip,deflate}
*
url_index.headers {credentials=myusername,mypassword}
url_index.headers {credentials=pedrobmat,bmatbmat}
*
url_index.headers {accept=text/xml|contenttype=text/xml;charset="utf-8"}
url_index.headers {postdata='index_variable_element'}

scope.range {(urlindex)|end}
** timespan calculation to enable to add the requested timespan from the config
index_variable_element.modify {calculate(format=F1)|'config_timespan_days' 1 +} * add 1 day because config_timespan_days is 0 based
index_variable_element.modify {calculate(format=timespan,hours)} * convert to the proper timespan string required for index_temp_3
index_temp_1.modify {calculate(format=date,yyyy-MM-dd)|'urldate'}
index_temp_3.modify {calculate(format=date,yyyy-MM-dd)|'urldate' 'index_variable_element' +}
index_variable_element.modify {clear} * clear the timespan value
index_variable_element.modify {addstart|'index_temp_1'T00:00:00Z'index_temp_3'T00:00:00Z}
end_scope

index_showsplit.scrub {regex||||} * copies the whole index page
scope.range {(splitindex)|end}
* now break up the large chuck into 3 smaller chucks, to speedup parsing
index_variable_element.modify {clear}

global_temp_1.modify {substring(type=regex)|'index_showsplit' "^.*?(.*?)"} * contains all programs
global_temp_1.modify {replace|title>|t>}
global_temp_1.modify {replace|subt>|st>}
global_temp_1.modify {replace||}
global_temp_1.modify {replace|syndicatedEpisodeNumber>|sEN>}
global_temp_1.modify {replace|description>|d>}
global_temp_1.modify {replace|originalAirDate>|oAD>}
global_temp_1.modify {replace|showType>|sT>}
global_temp_1.modify {replace|series>|s>}

** SPEEDUP option: if you don't need production/crew/cast info, just comment out next line
global_temp_2.modify {substring(type=regex)|'index_showsplit' "^.*?(.*?)"} * contains all productionCrew
global_temp_2.modify {replace|name>|>}
global_temp_2.modify {replace|given>|g>}
global_temp_2.modify {replace|sur>|s>}
global_temp_2.modify {replace|member>|m>}
global_temp_2.modify {replace|role>|r>}
global_temp_2.modify {replace|r>Actor|r>A}
global_temp_2.modify {replace|program=|p=}

** SPEEDUP option: if you don't need extra genre info, just comment out next line
global_temp_3.modify {substring(type=regex)|'index_showsplit' "^.*?(.*?)"} * contains all programs
global_temp_3.modify {replace|genre>|g>}
global_temp_3.modify {replace|class>|c>}
global_temp_3.modify {replace|relevance>|r>}
global_temp_3.modify {replace|programGenre>|pG>}
global_temp_3.modify {replace|(.*?)"} * contains all shows we need
index_temp_2.modify {substring(type=regex)|(]* station=\''config_site_id'\' [^>]*>)}
index_temp_2.modify {cleanup(removeduplicates=equal,100)}
index_showsplit.modify {set|'index_temp_2'}

end_scope
**
index_start.scrub {regex||time='[^\']*?T(\d{2}:\d{2}):\d{2}Z'||}
index_duration.scrub {regex||duration='PT(\d*H\d*)M||}
index_temp_5.scrub {regex||program='([^\']*)||}
index_videoquality.scrub {regex||hdtv='([^\']*)||}
index_videoquality.modify {replace(not="")|'index_videoquality'|HDTV}

index_subtitles.scrub {regex||closeCaptioned='([^\']*)||}
index_subtitles.modify {replace(not="")|'index_subtitles'|true}

scope.range {(indexshowdetails)|end}
index_start.modify {calculate(format=utctime)}
index_duration.modify {replace|H|:}

** get the programs part
index_temp_4.modify {substring(type=regex)|'global_temp_1' "^.*?

(.*?)

"}
index_title.modify {substring(type=regex)|'index_temp_4' "([^([^([^([^([^([^([^(\**)[\+]*"} * full stars
index_temp_1.modify {substring(type=regex)|'index_temp_4' ".*(\+)"} * half star
index_starrating.modify {calculate(not="" type=char format=F0)|#}
index_starrating.modify {addend('index_temp_1' not="")|.5}
index_starrating.modify {addend(not="")| / 4}
index_category.modify {substring(type=regex)|'index_temp_4' "([^(.*?)"}
index_actor.modify {substring(type=regex)|'index_temp_4' "[^A(.*?)"}
index_actor.modify {cleanup(tags="")}
*index_temp_1.modify {substring(type=regex)|'index_temp_4' "[^Guest Star(.*?)"}
*index_temp_1.modify {cleanup(tags="")}
*index_temp_1.modify {addend(not "")| (Guest Star)}
*index_actor.modify {addend('index_temp_1' not "")|'index_temp_1'\|}
index_actor.modify {cleanup(removeduplicates)}
index_presenter.modify {substring(type=regex)|'index_temp_4' "[^Host(.*?)"}
index_presenter.modify {cleanup(tags="")}
index_director.modify {substring(type=regex)|'index_temp_4' "[^Director(.*?)"}
index_director.modify {cleanup(tags="")}
index_producer.modify {substring(type=regex)|'index_temp_4' "[^Producer(.*?)"}
index_producer.modify {cleanup(tags="")}
index_producer.modify {substring(type=regex)|'index_temp_4' "[^Executive Producer(.*?)"}
index_producer.modify {cleanup(tags="")}
*index_temp_1.modify {substring(type=regex)|'index_temp_4' "[^Executive Producer(.*?)"}
*index_temp_1.modify {cleanup(tags="")}
*index_temp_1.modify {addend(not "")| (Executive Producer)}
*index_producer.modify {addstart('index_temp_1' not="")|'index_temp_1'\|}
index_producer.modify {cleanup(removeduplicates)}
index_writer.modify {substring(type=regex)|'index_temp_4' "[^Writer(.*?)"}
index_writer.modify {cleanup(tags="")}
* get the genres part
index_temp_4.modify {substring(type=regex)|'global_temp_3' "^.*?(.*?)"}
index_temp_1.modify {substring(type=regex)|'index_temp_4' ".*?(.*?)"}
index_category.modify {addstart('index_temp_1' not= "")|'index_temp_1'\|}
end_scope

** _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
** ##### CHANNEL FILE CREATION (only to create the xxx-channel.xml file)
**
* @auto_xml_channel_start
index_site_id.scrub {regex||()||}
scope.range {(channellist)|end}
index_site_channel.modify {substring(type=regex)|'index_site_id' "(.*?)"}
index_site_id.modify {substring(type=regex)|'index_site_id' ""}
index_site_id.modify {cleanup(removeduplicates=equal,100 link="index_site_channel")}
end_scope
* @auto_xml_channel_end

raedwa01
Offline
Joined: 6 years
Last seen: 6 years

I am getting the exact same error. I also downgraded to the base 2.0 and still am getting the error. I do see a 404 not found error at the bottom of the log...

MrGrymReaper
Offline
Has donated long time ago
Joined: 10 years
Last seen: 4 years

When you come up with a solution please contact me! As I too would like to be using 2.1 or above!

MrGrymReaper
Offline
Has donated long time ago
Joined: 10 years
Last seen: 4 years

This is through the JSON Schedules Direct I mean. If Webgrab++ doesn't support this I feel it would be good to add this. Since through schedules direct there wouldn't be any need for website scraping for the service. Along with its international service and the large amount of data available.

If it also is because WebGrab++ doesn't support JSON then it is also an important addition as from a post processing point of view "TheTVDB" only supports JSON REST based APIs and the same for "The Movie DB (TMDB)".

Blackbear199
Online
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 41 sec

SD(original ini) will be working again with the next release(have no ide when that will be).

as for the json service i have it workin with the combination of a php file as alot things happen.
i dont have it added yet to be able to add/remove lineups but using mx2xml to add a lineup i can grab the data with wegrab.

Blackbear199
Online
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 41 sec

wg 2.1.9 beta was released yesterday that has this fixed,see the change log for full list of changes.

sumwhereWA
Offline
Joined: 5 years
Last seen: 5 years

Hello I just started using this script and got it to grab from schedulesdirect, however it does not pull any tvguide data, just the channels:

[ ]
[ ] WebGrab+Plus/w MDB & REX Postprocess -- version V2.1.7
[ ]
[ ] Jan van Straaten
[ ] Francis De Paemeleere
[ ]
[ ] thanks to Paul Weterings and all the contributing users
[ ] --------------------------------------------------------------------------------
]
[ ] Job started at 28/05/2019 07:33:01
[ Debug ]
[ Debug ] Running on: Unix 4.18.20.0
[ Debug ] Environment: 4.0.30319.42000
[ Debug ] Mono version: 5.20.1.19 (tarball Thu Apr 11 09:09:21 UTC 2019)
[ Debug ]
[ Debug ] Loading timezone data
[ Debug ] Embedded timezones source: timezone.timezonesdata.txt
[ Debug ] Reading config file: /config/WebGrab++.config.xml
[ Debug ]
[ Info ] found: /config/siteini.pack/International/schedulesdirect.org.ini -- Revision 03
[ Info ] input file /data/guide.xml not found ... created a new one ...
[ Info ]
[ Info ]
[ Info ] i=index .=same c=change g=gab r=replace n=new
[ Info ]
[ Info ]
[ Info ] Group (0) :
[ Info ] update requested for - 0 - out of - 45 - channels for 6 day(s)
[ Info ]
[ Info ] Group (1) :
[ Info ] update requested for - 45 - out of - 45 - channels for 6 day(s)
[ Debug ]
[ Info ] ( 1/45 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=KOMODT (KOMO-DT)) -- mode Incremental
[ Info ]
[ Debug ]
[ Info ] ( 2/45 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=KINGDT (KING-DT)) -- mode Incremental
[ Info ]
[ Debug ]
[ Info ] ( 3/45 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=KCTSDT (KCTS-DT)) -- mode Incremental
[ Info ]
[ Debug ]
[ Info ] ( 4/45 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=KCTSDT2 (KCTS-DT2)) -- mode Incremental
[ Info ]
[ Debug ]
[ Info ] ( 5/45 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=KIRODT (KIRO-DT)) -- mode Incremental
..............

[ Info ] ( 45/45 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=KCTSDT4 (KCTS-DT4)) -- mode Incremental
[ Info ]
[ Debug ]
[ Debug ] 0 shows in 45 channels
[ Debug ] 0 updated shows
[ Debug ] 0 new shows added
[ Info ]
[ Info ]
[ ] Job finished at 28/05/2019 07:33:05 done in 3s
[ Debug ] statistics upload error: The remote server returned an error: (404) Not Found.

not sure what else to provide, but will be excited when I get this working.
If you need additional info, please let me know

sumwhereWA
Offline
Joined: 5 years
Last seen: 5 years

Nevermind...just noticed I was running an outdated version. I'll have to get the docker maintainer to update.

JetiMaster
Offline
Donator
Joined: 5 years
Last seen: 3 years
Blackbear199 wrote:

SD(original ini) will be working again with the next release(have no ide when that will be).
as for the json service i have it workin with the combination of a php file as alot things happen.
i dont have it added yet to be able to add/remove lineups but using mx2xml to add a lineup i can grab the data with wegrab.

Can u share your site ini for json service? Now that tvtv.us isn't working i think sd is the best option to grab data.

mat8861
Offline
WG++ Team memberDonator
Joined: 9 years
Last seen: 7 hours

Site.ini is available in site.ini pack, tested and works fine (you need user and password on scheduledirect).

EPGFreak
Offline
Donator
Joined: 5 years
Last seen: 2 weeks
mat8861 wrote:

Site.ini is available in site.ini pack, tested and works fine (you need user and password on scheduledirect).

What folder is it in? I am using site zip generated on 12.11.19.

mat8861
Offline
WG++ Team memberDonator
Joined: 9 years
Last seen: 7 hours
JetiMaster
Offline
Donator
Joined: 5 years
Last seen: 3 years

Thats the old one. We were talking about the new JSON one. =D

Log in or register to post comments

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl