New Version [BR] lineup.net.br

jeanniquini wrote:

Hi folks,
I was able to extract the new codes from the site. I also had to modify a word from the url in the .ini file.

Valeu muito obrigado, ficou perfeito.

jeanniquini wrote:

Hi folks,
I was able to extract the new codes from the site. I also had to modify a word from the url in the .ini file.

Como você extrai os código do site ?

ts easy to do if they do not change channel id's dynamically like they did before.this is what they used to do,i have no idea if they still do this as one would need to check the site data multiple time per day or even daily.

a good sign they are not is if this ini still works daily for a week.

google translate..

es fácil de hacer si no cambian dinámicamente la identificación del canal como lo hicieron antes. Esto es lo que solían hacer, no tengo idea si todavía lo hacen ya que uno tendría que revisar los datos del sitio varias veces al día o incluso a diario .

una buena señal de que no lo son es si esta información todavía funciona diariamente durante una semana.

before channel id's were changing every few hours if i remember right.

antes de que los ID de los canales cambiaran cada pocas horas, si recuerdo bien.

Hello sir.

I trying to run but its return this xml this guide.xml them I has attached.

I just set the channel "Cultura", but do not show the programation guide :-(

Somebody can help-me?

read my post above.i explained why it dont work...the chanel id change...

Can you give me the link or title of your post?

there is none.i am just explaining the problem.

when u have the correct channel id that good now(i got from the site)

update requested for - 1 - out of - 1 - channels for 1 day(s)
( 1/1 ) LINEUP.NET.BR -- chan. (xmltv_id=Cultura) -- mode Force
innnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
11.9 sec/update

so u need to dynamitically get this channel id as it changes..

jeanniquini wrote:

I did not know what the site did previously. It's been 4 days since the codes are still working. Let's wait for more days to see if they will change. I hope they continue the same.

based on that i dont think they are changing the channel id's anymore.it wouldnt still be working after that many days before.

i just check Cultura channel id now and its 636748,this should be the same as ur site_id="xxx" value?

Aumentar o timeout resolveu. Obrigadi jianiquini.

Thank you jianiquini

Jianiquini, tentei setar este epg na xtream, mas não lista nenhum canal quando vou configurar.

Pode verificar se cometi algum erro?

( 1/537 ) LINEUP.NET.BR -- chan. (xmltv_id=A&E) -- mode Force
i
no shows in indexpage!

( 2/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Agro Canal) -- mode Force
i
no shows in indexpage!

( 3/537 ) LINEUP.NET.BR -- chan. (xmltv_id=AgroBrasil TV) -- mode Force
i
no shows in indexpage!

( 4/537 ) LINEUP.NET.BR -- chan. (xmltv_id=AlJazeera) -- mode Force
i
no shows in indexpage!

( 5/537 ) LINEUP.NET.BR -- chan. (xmltv_id=AmazonSat) -- mode Force
i
no shows in indexpage!

( 6/537 ) LINEUP.NET.BR -- chan. (xmltv_id=AMC) -- mode Force
i
no shows in indexpage!

( 7/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Animal Planet) -- mode Force
i
no shows in indexpage!

( 8/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Arte 1) -- mode Force
i
no shows in indexpage!

( 9/537 ) LINEUP.NET.BR -- chan. (xmltv_id=AXN) -- mode Force
i
no shows in indexpage!

( 10/537 ) LINEUP.NET.BR -- chan. (xmltv_id=BabyTV) -- mode Force
i
no shows in indexpage!

( 11/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Bah!) -- mode Force
i
no shows in indexpage!

( 12/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band) -- mode Force
i
no shows in indexpage!

( 13/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Amazonas) -- mode Force
i
no shows in indexpage!

( 14/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Aracaju) -- mode Force
i
no shows in indexpage!

( 15/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Bahia) -- mode Force
i
no shows in indexpage!

( 16/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Brasília) -- mode Force
i
no shows in indexpage!

( 17/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Campinas) -- mode Force
i
no shows in indexpage!

( 18/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Curitiba) -- mode Force
i
no shows in indexpage!

( 19/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Maceió) -- mode Force
i
no shows in indexpage!

( 20/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Maranhão) -- mode Force
i
no shows in indexpage!

( 21/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Minas) -- mode Force
i
no shows in indexpage!

( 22/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Natal) -- mode Force
i
no shows in indexpage!

( 23/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Paulista) -- mode Force
i
no shows in indexpage!

( 24/537 ) LINEUP.NET.BR -- chan. (xmltv_id=Band Piauí) -- mode Force
i
no shows in indexpage!

Thanks for the return friend, I found it strange why the id is correct.

Do you know another website to use? used the net combo more delay around 6 hours and does not take the epg of all the channels

Group (0) :
update requested for - 170 - out of - 170 - channels for 2 day(s)
( 1/170 ) MEUGUIA.TV -- chan. (xmltv_id=A&E [Brazil]) -- mode Force
i
error downloading page: A conexão subjacente estava fechada: Erro inesperado em um envio.
pausing 1 of 4 times for 10 seconds before re-try.
error downloading page: A conexão subjacente estava fechada: Erro inesperado em um envio.
pausing 2 of 4 times for 20 seconds before re-try.
error downloading page: A conexão subjacente estava fechada: Erro inesperado em um envio.
pausing 3 of 4 times for 30 seconds before re-try.
error downloading page: A conexão subjacente estava fechada: Erro inesperado em um envio.
pausing 4 of 4 times for 40 seconds before re-try.
Unable to update channel A&E [Brazil]
Generic syntax exception:
message:
no index page data received from A&E [Brazil]
unable to update channel, try again later
No guide data to restore

Group (0) :
update requested for - 170 - out of - 170 - channels for 2 day(s)
( 1/170 ) MEUGUIA.TV -- chan. (xmltv_id=A&E) -- mode Force
i
error downloading page: A conexão subjacente estava fechada: Erro inesperado em um envio.
pausing 1 of 4 times for 10 seconds before re-try.
error downloading page: A conexão subjacente estava fechada: Erro inesperado em um envio.
pausing 2 of 4 times for 20 seconds before re-try.
error downloading page: A conexão subjacente estava fechada: Erro inesperado em um envio.
pausing 3 of 4 times for 30 seconds before re-try.
error downloading page: A conexão subjacente estava fechada: Erro inesperado em um envio.
pausing 4 of 4 times for 40 seconds before re-try.
Unable to update channel A&E
Generic syntax exception:
message:
no index page data received from A&E
unable to update channel, try again later
No guide data to restore
( 1/170 ) MEUGUIA.TV -- chan. (xmltv_id=Amazon Sat) -- mode Force
i
error downloading page: A conexão subjacente estava fechada: Erro inesperado em um envio.
pausing 1 of 4 times for 10 seconds before re-try.

can you send me your ini file and WebGrab ++ .config?

Thanks, can you send me the ini? because it continues with the same error as:

Group (0):
update requested for - 1 - out of - 1 - channels for 3 day (s)
(1/1) MEUGUIA.TV - chan. (xmltv_id = Baby TV) - mode Force
i
error downloading page: The underlying connection was closed: Unexpected error in a send.
pausing 1 of 2 times for 15 seconds before re-try.
error downloading page: The underlying connection was closed: Unexpected error in a send.
pausing 2 of 2 times for 30 seconds before re-try.

unfortunately the same mistake continues, did you know informs why?

WebGrab+Plus/w MDB & REX Postprocess -- version V2.1

Jan van Straaten
Francis De Paemeleere

thanks to Paul Weterings and all the contributing users

thank you was the version, if you need anything just talk to me.

So, I recently started using WebGrab+Plus to look at the EPG for my local channels and a lot of my local channels are only available on lineup.tv.br. As everybody in here commented already, the code ids for the channels keep changing on the website, which means I have to manually change the codes on the .ini whenever i want to retrieve them, and as I do them, they work.

Is there any pattern to the change? Are the numbers always random? I was thinking about creating a python script to look up the channel values and generate the .ini file for me on each run and running it before WebGrab+Plus, so it can automatically update the values - anybody did anything similar? I was trying to understand the .ini file, to see if some sort of search on the .html file could be performed in it, but I didn't quite understand how it works.

Also, I'm importing this EPG into TVHeadEnd, and TVHeadEnd has a limitation in regards to the program category - it's limited to 8 categories, not really dealing well with sub categories. I'm working on a script/setting for the .ini to replace the category names into the names supported by tvheadend, but I also wanted to ask if anybody was working on anything similar.

I did that already. In my region (Santos), there's 2 local rebroadcasters (Record TV Litoral, TV Tribuna Santos) and a local channel (Ideal TV) that are only available on mi.tv (donation-based) or LineUp.tv.br. On top of that, 2 local channels (ISTV and TV Unisantos) and 2 re-broadcasters (VTV, a SBT reprboadcaster, and TVB BAND, a BAND rebroadcaster) are only available on LineUp.tv.br and no other service, so donating for mi.tv would only partially solve my issue and no other provider has those local channels.

But that's ok - I created a Python3 script that will scrub the main LiveUp.tv.br website and acquire the changing identifiers for the channels. I edited the "run.sh" to run that before running the EPG grabber, so it generates a new WebGrab++.configuration.ini on every run. I'll keep using it for a couple days to test it.

Now, my next research topic will be Categories. WebGrab++ does scrub the categories of LiveUp.tv.br, but they're in Portuguese. TVHeadEnd (the software I import the .xml guide to) is not really compatible with different language categories, so I need to replace them with specific terms. I'll do a little more research and come back whenever I get that done.

I don't use WebGrab++ for IPTV services, I use it for my free OTA channels that I tune with an external antenna.

My setup is a little crazy, and I'm not sure this is the place to discuss it, but let's do it anyway (hope that's fine):

I have a Raspberry Pi running TVHeadEnd that tunes the OTA channels. It also scrubs the EPG from that broadcast, but for reasons I don't understand (I think either TVHeadEnd doesn't like the Brazilian EPG or the local broadcasters corrupt the EPG somehow), this EPG is quite incomplete/not tagged properly, so a lot of sections of it are left blank.

The Raspberry Pi also runs a script that allows Plex Media Server to tune TVHeadEnd channels. Since PLEX doesn't really support good Brazilian EPGs, I need to provide it with the EPG from a different source, and the one generated by the OTA EPG is filled with blanks, which in turn makes those channels untunnable on PLEX, as it doesn't know that there's a show happening (it's a dumb limitation, but that's how it is).

This is where WebGrab++ comes in: I scrub the EPG from the websites, import it to TVHeadEnd, which in turn serves it for PLEX, which makes it work alright (result in the attached file).

tl;dr: Rpasberry Pi runs TVHeadEnd, TVHeadEnd grabs EPG from WebGrab++, TVHeadEnd serves both Video and EPG to Plex Server.

The problem lies with the Categories: TVHeadEnd only really understands 10 specific categories and it's a text match. Since all the categories provided by LiveUp.TV.br are in portuguese, TVHeadEnd doesn't really understand them and assigns no categories to its internal EPG, which in turn is given to Plex, which makes plex not know what is a movie, what is a tv show, so on and so forth. (This is not a WebGrab++ issue, its parsing the categories as it should)

I can greate a siteini.user file for LiveUp.tv.br and add a "index_category.modify {replace(type=regex)|"^Ação$"|Movie / drama}" for instance, changing the categories as necessary, but this will create too many categories that TVHeadEnd will misinterpret - I need to study the relation between the unique categories in Portuguese to the valid categories in English, which I will do at a later date.

I'll come back with questions if necessary.

English please ....

mat8861 wrote:

English please ....

Sorry. I already edited my messages and translated them to English.

I didn't know the Forum was a English-specific forum, but I do agree that everything should be discussed in English - that way, everybody can pitch in and help. Brazilians usually revert to Portuguese, even in public forums like these, and I don't actually like doing that, but since I was spoken to in Portuguese, I thought this was allowed here.

Thank you for your help!

It's me again.

Like I said earlier, i'm trying to fix up linetv.TV.br because, for me specifically, it's quite useful.

I'm trying to parse the Rating of the shows. But i'm having a hard time understanding the .ini parsing language. I managed to fix the "episode number" (I just had to replace the "element end" from to ", but the rating I can't seem to figure out:

PS: I know people don't really care that much about this particular scraper, as everybody seems to be using the claro paid one, specially since most people use it for IPTV services, which doesn't offer local channels, but I would like to make this work, at least for my local channels, so any help would be appreciated!

look at the data more closely,there is no quotes(see screenshot).
I have no idea what the modify is your talking about,maybe this?
index_rating.modify {cleanup(style=upper)}

all this does is convert any lower case letters to upper(l to L for example).
your could also scrub the entire src=xxx as the index_ratingicon
not to many people use epg software that show it but the element does exist,this would display the actual gif.

Thanks, I feel like an idiot :D I've been debugging this code and re-writing it for the past 3 days: The "inspect" option on Chrome puts all links around quotes, even when they're not. It would never match.

I managed to put the icon, as you said, and the rating itself (using the number of the filename). I also noticed that, apparently, the "sub-title" aspect of all these programs is concatenated with the title as : (for instance, "Corujão: Carga Explosiva 3"). This is currently being throw into "title". I want to scrub that into title and subtitle and I mostly managed to do it with:

https://i.imgur.com/GNeijra.png

However, the search pattern is still hitting when there's no ":" on the title and putting the entire title into both, like so:

https://i.imgur.com/gVifNFA.png

But it works properly when there is a ": ":

https://i.imgur.com/3qRDSzT.png

I imaigne that WebGrab++ is not finding the divisor on the string, so everything is treated as first and/or last. Hrmm.

Another thing I don't quite understand: There's two .gifs for the channel logo on the page (one from the local broadcaster and another one for the network). The current scrubbing mechanism, using the "url" catcher (not the single or multi) grabs both and I can't figure out a way to limit it to one. I can't figure out how the "url" work.

Sorry for bothering you, blackbear, with these technicalities. I just figured I would try to at least get this up to date for my personal use, as nobody else would :D

when u use the separator method wegbgarb does exactly as you say.if it exists its used.if it don't it gets whats there.
easy solution is after ur subtitle.scrub add..
index_subtitle.modify {clear('index_title')}

for as the channel logo goes I don't understand what uo mean by 2 logo as I only see one.i am viewing the history channel(see screenshot).
also the url method works like this.

index_urlchannellogo {url|leadstring|scrub}

leadstring can be any web address the is present before the scrub part.
scrub is a regular serarator string scrub

when there is no leadstring it still must be specified as empty..
index_urlchannellogo {url||scrub}

also the url method is the same as using the element.scrub {single|bs|es|ee|be} so even if there were more that one logo is would only keep the first match.

take a look at the capture.JPG,both do the exact samething.one uses a leadstring and the other doesn't.

use view page source not inspect,i use chrome and this is where my screenshots are from.

inspect ==> opens page in dev tools
view page source ==> opens page in browser

I'm not managing to explain myself properly, I'll try to be a little more descriptive this time

Blackbear199 wrote:

when u use the separator method wegbgarb does exactly as you say.if it exists its used.if it don't it gets whats there.
easy solution is after ur subtitle.scrub add..
index_subtitle.modify {clear('index_title')}

I know this is a very specific issue, but here's a specific scenario of what I mean:

There's a TV Show called "Corujão" and that TV Show is actually just movies. So, the description for that TV Show, that's being scrubbed into the "title" key is "Corujão: De Repente é Amor"
What I want to do is scrub the title for : and split it into title and subtitle
Title: Corujão
Subtitle: De Repente é Amor

And that is working on my current setup, using a separator and the character ": " as the separator: It puts whatever is on the left on title (as I do include=first on title scrubbing) and whatever is on the right as sub-title (as I do include=last on subtitle scrubbing). What is happening is that, whenever there's a title that has no ": " in it, the entirety of the title is being inserted on both index.sub-title and index.title (I imagine because it can't find the separator).

What I really need, honestly, is a test pattern. if there's a ": " then do this, otherwise leave it all alone. With that, I can also parse the other shows from a different channel that do " - " instead of ": ". There's a couple of .ini files that have a test argument that i need to look more into.

Blackbear199 wrote:

for as the channel logo goes I don't understand what uo mean by 2 logo as I only see one.i am viewing the history channel(see screenshot).
also the url method works like this.
index_urlchannellogo {url|leadstring|scrub}
leadstring can be any web address the is present before the scrub part.
scrub is a regular serarator string scrub
when there is no leadstring it still must be specified as empty..
index_urlchannellogo {url||scrub}
also the url method is the same as using the element.scrub {single|bs|es|ee|be} so even if there were more that one logo is would only keep the first match.
take a look at the capture.JPG,both do the exact samething.one uses a leadstring and the other doesn't.
use view page source not inspect,i use chrome and this is where my screenshots are from.
inspect ==> opens page in dev tools
view page source ==> opens page in browser

You are correct, on the example you used. Sometimes (TV Tribuna is an example, currently available on http://www.lineup.tv.br/gdc.php?Guia=922704) there's two images:

https://i.imgur.com/xwlvT7G.png

Which then created a .xml file with two icons, like so:

src="http://www.lineup.tv.br/canais/tvtribuna.gif|http://www.lineup.tv.br/canais/globo.gif"

I tried moving the url method around but couldn't get a positive response. I'll try more now that I have a better understanding of how it works. Thanks!

in this case I wouldn't use url method(see screenshot).
this will use the first logo when there is more than one.

regarding the title/subtitle..
using the separator method as your doing will get the correct results when a : exists.
do u understand what the modify does?
index_subtitle.modify {clear('index_title')}
if..
index_title and index_subtitle are different they are left alone as in the case when the data has the :
index_title and index_subtitle are the same as in the case when there is no : ==> subtitle gets removed.

also u can use multiple separators..
separator=":""-"
u can have as many as u want,just encase each in its own set of quotes.

Interesting, I better understood the modify element and managed to use it in other areas of the xml at this point. However, I'm getting stuck in two different issues:

To simplify and streamline my "title" scrubbing, I decided to simply scrap the entirety of the title and work on it based off the "category" that the program is classified as, if necessary: Right now this helps me immensely and makes everything simpler. Here's a snippet of the "title" and "subtitle" assignments based off a single category named "Filme". Screenshot is commented for further clarification:

https://imgur.com/a/8dLfg58

What I want to do is add, on that same "modify set" line, different categories. Adding "\|"Cinema"" and "\| 'index_category' == "Cinema"" didn't seem to work, it would just perform the temporary assignment to every single program. I'm missing something on the "or" for the test argument.

Not only that, on the separator/include/exclude i'm having a problem: It seems like the (what i assume is) underlying regex on WebGrab++ is identifying all instances of ": " in the same selection, making a title with multiple ": " like this:

https://imgur.com/hMcbyw6

be parsed with two titles, like this:

https://imgur.com/Nbd3U75

Ideally, what I would prefer, is that the identification of the separator to stop after the first hit. I either need that or a different way to perform the same check.

I'm currently not cleaning up any episode information from the title on purpose, as I need to better understand the episode.scrub statement on the current .ini. I'm not a good programmer and REGEX has always been a pain in my ass (and i'm trying to understand it by using regex101.com, a live parser), but that's for another day :D

ur beyond my help without me typin a few pages why.
read the manual,all ur answers r there.

Ok, so, I went through the file and pretty much touched everything. Here's an incomplete list of changes I made:

- Changed dateurl
- Changed the way Icons are acquired, to only scrub the first Icon from channels that have more than one
- Subtitle scrubbing
- Title/Subtitle conditioning based on program (For easier movie identification)
- Better Episode scrubbing
- "original-air-date' creation
- Many fixes, mostly focused on Plex usability.

Everything is commented to the best of my abilities. I've been using it for the past 3 days and haven't noticed anything too broken, so I decided to release it as is. At this point i'm pretty much done, as the file is serving all my requirements. I decided to upload it here so people can improve on it, if necessary, and to get a bigger sample size (which I find doubtful, as the Channel IDs keep changing). Any help and/or comments will be greatly appreciated.

The only way I managed to fix the problem of the ever changing channel IDs was by using a custom Python3 script before WebGrab++ runs (it's just a call made on run.sh before it starts - it works) so it updates the WebGrab++.config file. However, the script only updates the channels you tell it to, by using the channel names as the search string - I can post that too if necessary, but it's not the cleanest of scripts and I'm not updating it to be a more general one.

Also, it seems that the tvlineup website has the weird thing of using past days and moving them to the next day, starting from Sunday.
It list shows from Sunday-Saturday, and when it moves up a day (showing Monday-Sunday shows), it replaces the "next" Sunday with the programming from the previous Sunday. This behavior only stops on the next Sunday, when it will again show 6 days of programming correctly. I don't how how to fix that other than only scraping the website on a Sunday.

Thanks for all the help and sorry for the constant messages.

ur thinking along the right lines,i will give you a few clues but not the answer.

as you know the guia=xxxxx id keeps changing.
but the channel name doesn't.
on the gdc.php page(i think) there is a list of all the channels.

so what u want todo is use the channel name for the site_id="xxxxx" and not the channel id value as it changes.
then u can search gdc.php in real time using ur python script(i like php).
find the guia=xxxx value currently being used and bobs ur uncle so to speak(just remember ur regex or whatever u use matches only the channel u want epg for in your search string).

i wrote this in php few yrs ago and still works flawless.

latest beta version(not public) should also be able todo this without any external scripts,ect
no idea when the next rlease will be but i have read some docs,never tried it myself.

last hint..
urlencode as some channel names have accented character in their names.

Blackbear199 wrote:

ur thinking along the right lines,i will give you a few clues but not the answer.
as you know the guia=xxxxx id keeps changing.
but the channel name doesn't.
on the gdc.php page(i think) there is a list of all the channels.
so what u want todo is use the channel name for the site_id="xxxxx" and not the channel id value as it changes.
then u can search gdc.php in real time using ur python script(i like php).
find the guia=xxxx value currently being used and bobs ur uncle so to speak(just remember ur regex or whatever u use matches only the channel u want epg for in your search string).

Oh, I am fully aware. My script is working as we speak: it matches the names of the channels I want (from a variable I manually set on the script) to grab the EPG from, looks for the uid, creates a valid WebGrab++.config.xml with the actual site_ids and it works. It's just a quickly thrown together script because I didn't think of releasing it.

I'm using that .ini I posted there with my script running before webgrab and everything's been working fine.

Thank you for your help!

yes,u can do it multiple ways,if you have a bit knowledge.
u can either do it ur way or mine.
or even use a sh,bat,exe for ex.
webgrab can run these(see manual)

lookfor set(type=run)

Using a script in combination with webgrab will work as I'm using such method for a while. Basically I'm using php to scrap the channel ids from lineup and generate a WebGrab++.config.xml dynamically every day.
After that I call the WebGrab+Plus executable with some parameters pointing to another folder (that has site.ini, existing guide, etc) and wait a lot to grab everything.
Sometimes I got "No shows in index page" and this indicates to me that channelid has changed on their site so I run it again.
For my particular install I upload the generated guide.xml into a external webserver using rclone (just because it's easier for me).
In my setup I made some changes allowing Kodi to identify Season, Episode name and number when available because my setup is with Kodi + TvHeadend.

Hope it helps you.

here test it for 3 weeks. See if it works.
decryptkey=test