Mentioned in Tweet Scrape - Missing Mentions - Major Bug Report

[Rev]

Software:
Twitterdub

Version:
3.561

Location:
(Tab) Scraper > (Title) Scrape Profiles

Component:
(Function) Scrape Profiles

Description:
Doing a profile (URL) scrape with the "Custom Search" options of "A Tweet Id or Url" and "Mentioned in Tweet" results in missing mentions. For example, trying to scrape a tweet with 3 mentioned users/profiles results in just scraping 1/3 of them. No "User Filter" options are included.

Expected-Results:
Scraper to "Mentioned in Tweet" profile-scrape all the valid/available users in a tweet.

Attachments:

Code: Select all

Setup custom search controller
Custom search run: search: https://twitter.com/USERNAME/status/ID
A Tweet Id or Url-Mentioned in tweet
Perform custom search: 'A Tweet Id or Url' #chain/total: 1/2 using: USERNAME
Cannot request unlimited items per chained search: setting: 100
Start search: A Tweet Id or Url with: https://twitter.com/USERNAME/status/ID(100) using: USERNAME
Results of search: 1/100 A Tweet Id or Url with: https://twitter.com/USERNAME/status/ID
Handle results: 1 nextstep: 2/2
Got results from chain: 1 feed into next chained custom search item
Handle Result:https://twitter.com/USERNAME/status/ID  step: 1/2 Result: 1/1
Perform chained search: [1] https://twitter.com/USERNAME/status/ID
Perform custom search: 'Mentioned in tweet' #chain/total: 2/2 using: USERNAME
User specific num per stage to return: 3
Start search: Mentioned in tweet with: https://twitter.com/USERNAME/status/ID(3) using: USERNAME
Search: https://twitter.com/USERNAME/status/ID with: USERNAME: max: 1 return: RETURN_URLS
Searching page: 0 TweetDetails https://twitter.com/USERNAME/status/ID
found: @USERNAME
found: https://twitter.com/USERNAME/status/ID
Cursor: ID
Search pages: 0
Got enough: 1/1
Results: 1
No more pages required
TweetDetails: ID https://twitter.com/USERNAME/status/ID
Scraped users: 1
Results of search: 1/3 Mentioned in tweet with: https://twitter.com/USERNAME/status/ID
Handle results: 1 nextstep: 3/2
End of chain: Store results: 1
Finished search: results: 1
Results: 1

Submitted-Logs-ID:
logs_38110

[martin@rootjazz]

it's the comma , after the @mention causing the fail.

The next update will fix this. I shall let you know when it is ready.



Regards,
Martin

[martin@rootjazz]

https://rootjazz.com/twitterdub/updatetesting.html

[Rev]

I just retested this to see if everything works now and found out the module is still having issues and missing some mentions. The version of software that I'm using is the latest version (3.562) there is for the update-testing editions. I tried to "Mentioned in Tweet" scrape 4 more different subjects (out of a bunch) and they all failed to scrape all the available mentions in their respected tweets.

Here are the logs to my individual testings:

logs_20642
logs_54597
logs_27041
logs_34971

Also, here are my process-results for the mentioned logs, the "out of" numbers indicate how many mentions are missing after a completed scaping-process (succesfully-scraped / supposed-scrapes):

Logs#1 = 1/2
Logs#2 = 1/2
Logs#3 = 1/2
Logs#4 = 3/4

[martin@rootjazz]

I just retested this to see if everything works now and found out the module is still having issues and missing some mentions. The version of software that I'm using is the latest version (3.562) there is for the update-testing editions. I tried to "Mentioned in Tweet" scrape 4 more different subjects (out of a bunch) and they all failed to scrape all the available mentions in their respected tweets.

Probably the tweet has some non-exected chars after the @mention, I'll need to check the specific tweets where the @mentions were not all pulled. If you have examples, that would help a lot.

before program was failing to detect @profile,@profile2 with a comma separator char, probably something similar now too

[Rev]

I just retested this to see if everything works now and found out the module is still having issues and missing some mentions. The version of software that I'm using is the latest version (3.562) there is for the update-testing editions. I tried to "Mentioned in Tweet" scrape 4 more different subjects (out of a bunch) and they all failed to scrape all the available mentions in their respected tweets.

Probably the tweet has some non-exected chars after the @mention, I'll need to check the specific tweets where the @mentions were not all pulled. If you have examples, that would help a lot.

before program was failing to detect @profile,@profile2 with a comma separator char, probably something similar now too

The previous comma seperator character caused issue seems to be fixed as you have mentioned but, I can't tell if there are other possibilities with the same or different characters that causes this issue. I will be on this matter and keep on searching once I have time.

Would you like me to submit logs as usual or send you the faulty tweet-links via forum direct-messages? Both are fine by me.

[martin@rootjazz]

The previous comma seperator character caused issue seems to be fixed as you have mentioned but, I can't tell if there are other possibilities with the same or different characters that causes this issue. I will be on this matter and keep on searching once I have time.

Would you like me to submit logs as usual or send you the faulty tweet-links via forum direct-messages? Both are fine by me.

Just post a link to the tweet that doesn't scrape all @mentions.

Or if you want to keep it private, email me

Code: Select all

support[at]rootjazz[dot]com

Regards,
Martin