Import XMLTV files in TV-Browser

For talk and Help about everything concerning TV-Browser.
Benutzeravatar
bodo
Site Admin
Beiträge: 19635
Registriert: 03 Dez 2003, 19:37
Wohnort: Köln
Kontaktdaten:

Beitrag von bodo » 07 Jun 2007, 15:28

I know that some of the grabbers are legal, but there is no way for us to determine if the xmltv file is legal. E.g. the german xmltv-grabber was not legal the last time i checked. the developer of the grabber wrote me an email that he hasn't ask for permission and never wants to do that because he was afraid to get kicked from their servers.

Benutzeravatar
bodo
Site Admin
Beiträge: 19635
Registriert: 03 Dez 2003, 19:37
Wohnort: Köln
Kontaktdaten:

Beitrag von bodo » 07 Jun 2007, 15:38

Ps: You should remove all XMLTV-Grabber from the your distribution that are not 100% legal to show that you are willing to respect the copyright-laws in the different counties. Without that step I will continue to think of xmltv as illegal source.

Bananeweizen
Site Admin
Beiträge: 3353
Registriert: 09 Sep 2006, 22:45

Beitrag von Bananeweizen » 07 Jun 2007, 16:22

To make this discussion even longer: Bodo is completely right.

In Germany it is illegal to copy electronic information which is stored in a database or accessed in a database like way. There are enough decisions by court which explicitly state that this is also true for database information shown on web pages, even if a normal user can access the content of the whole database by repeatedly querying the web page. (For example there was a decision about a flight portal grabbing all the whether information from the "Deutscher Wetterdienst" website some days or weeks ago). So grabbing the TV data from a German web site is in fact illegal in Germany.
You may iterate again and again that only a lawyer might give such a statement and I as a developer may not, but that doesn't change the legal situation. And don't try to talk us into something, where you don't have any legal risk at all, but we have.

If I were an XMLTV developer, I probably wouldn't want to hear things like that, too. Especially if I put my spare time in such a project. And even more so if I only used other grabbers, which are competely legal. But have you even thought about the fact that your project might be more successful and might attract more developers and users if you clearly stated that you are willing to respect such copyright (or other legal) issues?

And yes, I know that from a technical point of view you may be able to add XMLTV data to TV-Browser (it's open source at the end) without us giving you any permission or technical information, but you really shouldn't try. In the end this would just lead to the TV-Browser project being involved in those legal discussions and probably not getting any more (legal) TV data from our current TV data providers, making the whole project useless.

Ciao, Michael.

PS: Personally I'd also like to use the XMLTV data as source for TV-Browser and I investigated the technical possibilities some time ago. But I stopped that evaluation because of the legal issues.

rmeden

XMLTV lead responds...

Beitrag von rmeden » 08 Jun 2007, 04:26

My name is Robert Eden, and I'm the acting leader of the XMLTV project.
I'd like to clear up a few misconceptions about XMLTV project being a den of illegal copyright violations..

XMLTV got it's start as just a file format. It was created to deal
with the problem where every data source had their own format. Instead
of applications having to deal with multiple formats, XMLTV provides a
common non-proprietary framework. Everybody wins.

The core XMLTV project then started including programs to collect and
process data in addition to a library of helpful tools. In addition,
there are many data sources and applications using the XMLTV data
format that are not part of the XMLTV project.

I personally wrote the U.S. grabber rseller would like to use.
(tv_grab_na_dd). It gets data not via a web scraper, but from a feed
provided by Tribune Media Services, one of two companies that
consolidate U.S. TV listings. (the other is TV Guide). TMS provides
the data as a service to the community for private, non-commercial use
as a research project. The legality of that grabber is beyond
question.

Yes, some grabbers are web scrapers. Historically the XMLTV project
has been very open about it's operations. We try to be good Net
Citizens.

The grabber author is supposed to verify that the source web page
allows or at least doesn't forbid in its terms of service the scraping
of data for non-commercial use. The XMLTV grabbers all identify
themselves with a XMLTV ident header. It is trivial for a webmaster to
block us and they can tell who are are from their logs. A scrapper is
removed from the distribution at the request of the data source or at
the first sign of intentional blocking. (Both have happened)

There may be other grabbers out there that use the XMLTV file format
and are not as respectful to their data sources. But they are not part
of the XMLTV distribution.

Your project doesn't need to support the XMLTV file format, but in
doing so, you're choosing to exclude a significant worldwide user base.

Even if some grabbers would be illegal under German law, they are not illegal under the laws of other countries. No one is asking you to provide a grabber, legal or not, or even to put potentially illegal files on your systems. You're simply being asked for help reading a data format.

If you need a file to test, feel free to use tv_grab_na_dd as it's legality is beyond doubt. (The copyright holder is giving the data directly to us using an interface they developed for that purpose.)

Look at the MP3 data format. You may be shocked to hear that some MP3s
contain copyright violations. That doesn't condemn the file format,
make everything stored as a MP3 illegal, or make every application that
uses MP3 illegal. It has many legal uses. Same with XMLTV. Do you have any MP3 files on your computer (legal of course).


Robert

Benutzeravatar
bodo
Site Admin
Beiträge: 19635
Registriert: 03 Dez 2003, 19:37
Wohnort: Köln
Kontaktdaten:

Beitrag von bodo » 08 Jun 2007, 05:33

Yes, XMLTV is a file format. But as I wrote earlier: the german grabber is not legal, because you don't have a permission to grab the source.

My personal opinion is that your aproach to this problem is wrong. You say: if they want they could stop us doing this grabbing very easily. This is not what a lawyer would advice you to do. You have to ask _before_ you start grabbing. Only with this way you ensure that your project is 100% legal.

You say that a author is supposed to very or at least doesn't forbit the scraping of data in their terms of service, but a Website doesn't have to forbit this behaviour explicitly. If the copyright law forbits this behavior, why should i add this to the terms of the webpage? You should inforce a policy that the author of the grabber has to insure you that the grabber is legal. Only legal grabbers should be added to your distribution.

And again: last time I checked, the tv_grab_de_tvtoday in _your_ distribution was not legal. I have a mail here in my inbox from the author of that grabber that says he hasn't ask the webpage-admin to use the data. I repeat: it's not a "other grabbers out there", it's a grabber you offer.

If we would add a possibility to add XMLTV to our Software, we would get a possible target for copyright laywers here in Germany. And that's a risk we won't take. Even if it will increase our userbase. Adding a bittorrent client with direct tv show download and sharing would increase our userbase, too, but we won't do that either ;). Even if the base-technology (sharing files) is legal, sharing copyrighted content is not. Same with your file format.

You should get the grabber legal under the law of the country the tv-channel belongs to and the datasource worsk in. That would be enough to make the project legal.
Zuletzt geändert von bodo am 08 Jun 2007, 05:57, insgesamt 1-mal geändert.

Benutzeravatar
bodo
Site Admin
Beiträge: 19635
Registriert: 03 Dez 2003, 19:37
Wohnort: Köln
Kontaktdaten:

Beitrag von bodo » 08 Jun 2007, 05:44

Ps.: our users can offer data for our system themselfs, but before their data gets listed in our software, each user has to prove that the data he offers was collected in a legal way. It's not that hard to do. We do it, why don't you?

Benutzeravatar
bodo
Site Admin
Beiträge: 19635
Registriert: 03 Dez 2003, 19:37
Wohnort: Köln
Kontaktdaten:

Beitrag von bodo » 08 Jun 2007, 06:29

I just checked the Terms and Conditions of TV Today:

"Verbreitung nur mit schriftlicher Genehmigung des Verlages. Dies gilt auch für die Aufnahme in elektronische Datenbanken und Vervielfältigung auf CD-ROM."

Found here: http://www.tvtoday.de/service/nutzungsbedingungen

Translated:
"Spreading only with written permit of the publisher. This also applys for inclusion in electronic databases and duplication on CD-ROM".

In german law a database is not only a sql-database, it could also be a simple XML-File (as seen in the Wetterkanal-Rouling). This makes it 100% clear that you really have to get a written permition. Even under your "not so legal" own rules ;).

Do you have it?

rmeden

Beitrag von rmeden » 08 Jun 2007, 06:44

When you say "before their data gets listed in their software", do you mean you merge everyone's data centrally (not locally like MythTV). I agree that doing so would in many cases be a copyright violation.

Are you looking to redistribute XMLTV (even XMLTV.EXE). If that's the case, again I see your point.

But if you're not doing either of those things, I don't see the exposure for providing a filter to read the data files (obtained via tools available elsewhere). But your BitTorrent analogy is a good one (similar to my MP3 analogy) and I respect your decision, even if I don't agree with it.

It sounds like German law is much stricter than other countries, even in the case of offering a tool that someone could abuse.

Part of the problem could be we both see XMLTV mainly as the grabber for our respective countries... I know my XMLTV grabber is legal, you feel (or even know) yours is not, and we both assume that if a filter is written it will be primarily used for legal or illegal use respectively.

If you do wish to support rseller and other U.S./Canada users with your application, you can write something to get data directly from labs.zap2it followed by com. (can't post a URL) You can use the XMLTV certificate code ("ZYZM-TE5O-SBUT") to create an account and look around. Zap2IT provides all the details on their XML format and includes sample programs to get the data (HTTP/SOAP). The _na_dd XMLTV grabber simply fetches the data and reformats it into XMLTV format.

Robert

rmeden

Beitrag von rmeden » 08 Jun 2007, 06:53

<useless reply removed>
Zuletzt geändert von rmeden am 08 Jun 2007, 07:07, insgesamt 1-mal geändert.

rmeden

Beitrag von rmeden » 08 Jun 2007, 07:05

FYI.. I just noticed the "Also applies to" part... I've started an email thread on the the XMLTV-Developers list about removing _de_tvtoday from the distribution as it does seem to violate the terms of service.

Robert

Benutzeravatar
bodo
Site Admin
Beiträge: 19635
Registriert: 03 Dez 2003, 19:37
Wohnort: Köln
Kontaktdaten:

Beitrag von bodo » 08 Jun 2007, 07:13

Sorry, but the important word is not spreading, it's "This also applys for inclusion in electronic databases ". This means this applies not only to spreading, but also to including into own databases, even for private use.

In Germany it's not allowed to grab data from a database like structure without a permission. Even for private use. It's in § 53 UrhG.

And yes, our solution is to contact the companies that offer the data and write a plugin that converts the data (e.g. this was done for radiotimes).

I know that you include grabber that are legal, but some are not. And it seems that you are not willing to remove them. This decission could jeopardize your complete project.

And as final note: i really like XMLTV and we as project would love to include a Plugin that reads the data. Infact we had a plugin that could do this until we got informed that this was risky. If would make a public decision against grabbing sources without permission, we would love to include the xmltv-plugin to our software.

Ps: sorry, you posted while i wrote this. It's good that you remove that source.But you really should ask every(!) datasource for a permission. At least to show some good will.

Bananeweizen
Site Admin
Beiträge: 3353
Registriert: 09 Sep 2006, 22:45

Beitrag von Bananeweizen » 08 Jun 2007, 13:50

In response to the discussion on the xmltv-devel list:
Single users are saving the information on their computers, but it can't be possible that TvToday can say:
"you are allowed to open our homepage, but not to save the data you
see" ... or even better: "you are allowed to save our homepage, but
only in html format, not in xml"...
.

Well, it's not exactly like that, but nearly. You are allowed to save the TvToday homepage (for instance by means of your browsers "Save page" function). And you may transform that data into other formats and print it or whatever. And you are also allowed to do this for several pages, because that's what a normal user will do when browsing the page.

(Another analogy for allowed use would be books: You can take a book, read each page, each word, each character. But you are not allowed to make copies of each character, each word or each page. You only got usage-rights for content, but no copy-rights).

But you are not allowed to request each page of TvToday and to create a copy of the underlying database information. It doesn't matter whether TvToday stores that as SQL database and you store it as XML file.
BTW: Even if the whole database content where on a single page (so you wouldn't need to grab repeatedly) you are not allowed to copy the database information (you are allowed to save the page nevertheless). The idea of this kind of legal construction is the experience that the preparation, ordering, unification, ... of structured information (i.e. the creation of that database) is much work (which also has higher value than just having all that unordered information) and shall therefore not be copied without permission.

Maybe a lawyer would kill me for explaining it this way, but the essence (with German law) is: A database (in whatever form) contains a lot of work for structuring that information and therefore has its own copyright (unrelated to the copyright of the information stored in the database).

Side note (unrelated to XMLTV and legal issues): We also do have the permission to use zap2it, but our tvdataservice for zap2it is out of order (it got not maintained for quite some time due to our main user base being in Europe). We are always looking for people who want to spend some time to bring TV-Browser to more countries (by translation or providing TV data).

rseiler

Beitrag von rseiler » 09 Jun 2007, 01:35

Bananeweizen hat geschrieben:Side note (unrelated to XMLTV and legal issues): We also do have the permission to use zap2it, but our tvdataservice for zap2it is out of order (it got not maintained for quite some time due to our main user base being in Europe). We are always looking for people who want to spend some time to bring TV-Browser to more countries (by translation or providing TV data).
Is this what Robert was talking about a few posts up, maybe adding support to TV-Browser to access Zap2it's service directly (not involving XMLTV)? That would seemingly be the most straightforward approach and would solve the problem of North America being excluded.

Benutzeravatar
bodo
Site Admin
Beiträge: 19635
Registriert: 03 Dez 2003, 19:37
Wohnort: Köln
Kontaktdaten:

Beitrag von bodo » 09 Jun 2007, 08:10

The (not working) zap2it Importer can be found here:

http://tvbrowser.svn.sourceforge.net/vi ... timporter/

But it needs a lot of work. The best would be to write a new importer. The current zap2it needed way to much ram.

An example for a working importer can be found here:

http://tvbrowser.svn.sourceforge.net/vi ... taservice/

or here:

http://tvbrowser.svn.sourceforge.net/vi ... taservice/

Jo
Site Admin
Beiträge: 10467
Registriert: 07 Apr 2006, 23:39

Beitrag von Jo » 10 Jun 2007, 10:57

XMLTV developer hat geschrieben:Sorry to bring back an old thread. But many of the XMLTV grabbers are infact legal. The US grabber is using Zap2It DataDirect, with permission. The Swedish and Norwegian grabber is using data directly from the tv-companies themselves, with permission.

I believe the UK grabber as well is using data with permission...

Christian...
(Norwegian Grabber developer)
There are two grabbers for Norway. "no_gfeed" might be legal, but the other one (which also contains few German channels) most likely has no permissions.

Gesperrt