Scrape Twitter - suggestion

Discussions to do with Soundcloud Manager. Do not use for support, use the dedicated support forum for help requests
Bartekef
Posts: 677
Joined: Thu Sep 22, 2016 12:24 pm

Re: Scrape Twitter - suggestion

Post by Bartekef »

martin@rootjazz wrote:https://rootjazz.com/listjazz/

now includes scrape all URLs and scrape all URLs containing "x"
Thank you Martin,
although link doesnt work for me:(
User avatar
martin@rootjazz
Site Admin
Posts: 34640
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Twitter - suggestion

Post by martin@rootjazz »

that is because it is not a link. I am sure you can figure out how to get the file downloaded, if not, then the app might not be a good fit for you as it is a console based app and requires text inputs :)
Bartekef
Posts: 677
Joined: Thu Sep 22, 2016 12:24 pm

Re: Scrape Twitter - suggestion

Post by Bartekef »

Damn haha yeah im total newbie when it comes to this kind of stuff

anyways, im eager to learn, and I know that many times you have to learn to overcome the obstacles, so, if you showed me the way how I could get these twitter url's in the easiest way then Im about to start it asap :)
Bartekef
Posts: 677
Joined: Thu Sep 22, 2016 12:24 pm

Re: Scrape Twitter - suggestion

Post by Bartekef »

martin@rootjazz wrote:that is because it is not a link. I am sure you can figure out how to get the file downloaded, if not, then the app might not be a good fit for you as it is a console based app and requires text inputs :)
Damn haha yeah im total newbie when it comes to this kind of stuff

anyways, im eager to learn, and I know that many times you have to learn to overcome the obstacles, so, if you showed me the way how I could get these twitter url's in the easiest way then Im about to start it asap :)
Bartekef
Posts: 677
Joined: Thu Sep 22, 2016 12:24 pm

Re: Scrape Twitter - suggestion

Post by Bartekef »

*even if I had to use that console stuff etc.
I could learn that
User avatar
martin@rootjazz
Site Admin
Posts: 34640
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Twitter - suggestion

Post by martin@rootjazz »

ok, so go to the link I sent

copy it
paste it into your browser
press enter
click ok to download the file

unzip the file
run the .exe

now follow instructions in the app
Bartekef
Posts: 677
Joined: Thu Sep 22, 2016 12:24 pm

Re: Scrape Twitter - suggestion

Post by Bartekef »

martin@rootjazz wrote:ok, so go to the link I sent

copy it
paste it into your browser
press enter
click ok to download the file

unzip the file
run the .exe

now follow instructions in the app
Yes I tested it and it worked for a while very well! Now I have a problem:

* FAILED: 98734895 : Couldn't get user from value:
Follow: http://twitter.com/#!/storezb_rlow using: xxx
* FAILED to get ID from : http://twitter.com/#!/storezb_rlow
* SKIP: Exception of type 'LibUtil.ExSkipItem' was thrown.
* IGNORE: http://twitter.com/#!/storezb_rlow

1. Why it failed to get ID from that link? When I open it in my browser, it works fine. You know, these URLs are from that program you given to me, scrapped from Soundcloud, and people write different urls on their SC bio's, sometimes the url has that '#!/profile_name', the other time its without '#!" but it has @ before profile name : '...twitter.com/@profile_name' ; like everyone put different twitter link in Soundclouds description. Maybe if SCM scraped every URL and make it without that '@' or "#!' that would solve the problem? You definitely would work it out, though
2. It followed 40 people today without a problem. After that, I got that ' SKIP: Exception of type 'LibUtil.ExSkipItem' was thrown.' on every follow attempt.

I deleted existing logs and performed new action.
LOGS: 19334_bartekfraczyk...

Lookin forward to hear from you!
Best
User avatar
martin@rootjazz
Site Admin
Posts: 34640
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Twitter - suggestion

Post by martin@rootjazz »

Why are you including #!

Yes twitter handles it, look at the URL, they remove it.

The URL as you provided is NOT a valid profile URL
Bartekef
Posts: 677
Joined: Thu Sep 22, 2016 12:24 pm

Re: Scrape Twitter - suggestion

Post by Bartekef »

martin@rootjazz wrote:Why are you including #!

Yes twitter handles it, look at the URL, they remove it.

The URL as you provided is NOT a valid profile URL

Im not including it. Do you remember adding scrape urls from profile to SCM for me? When I scrape them urls and then get these including 'twitter' using ListJazz, it gives me URLs with '#!', but also without it. Check this link: https://www.sendspace.com/file/lwqrwc
This is txt with twitter urls firstly scrapped with soundcloud manager, then extracted those urls including 'twitter' in it
User avatar
martin@rootjazz
Site Admin
Posts: 34640
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Twitter - suggestion

Post by martin@rootjazz »

ahhh SCM is scraping them!

well, easiest would be to do a search and replace on your list to remove them :)



Note made to take a look at why they are being scraped
Post Reply