Problem with 60s delay

User avatar
martin@rootjazz
Site Admin
Posts: 26208
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Problem with 60s delay

Post by martin@rootjazz » Sat Nov 21, 2020 3:03 pm

martin@rootjazz wrote:
Sat Nov 21, 2020 2:56 pm

Ok, this I CAN check on. Sounds like the GUI control is not loading correctly or the value is not stored. I will get back to you
Bug fixed for ACCOUNT form verification options.

The next update will fix this. I shall let you know when it is ready.



Regards,
Martin

Faz
Posts: 58
Joined: Thu Mar 15, 2018 12:42 am

Re: Problem with 60s delay

Post by Faz » Sat Nov 21, 2020 3:35 pm

martin@rootjazz wrote:
Sat Nov 21, 2020 2:57 pm
You can read issues with previous driver releases going back years that initial builds have some issues with chrome interaction due to big chrome changes.

It could / could not be the issue. Hard to say unless a new driver update comes out with an explanation that it fixes an issue with previous build
Yes, I didn't mean to make a big fuss & problem out of this. Just mentioned it to avoid, that you think there is something wrog specifically with the v87 chrome. IMO my problem with instagram opening in the chrome window do not seem to be related to this latest chrome version.

Faz
Posts: 58
Joined: Thu Mar 15, 2018 12:42 am

Re: Problem with 60s delay

Post by Faz » Sat Nov 21, 2020 4:02 pm

martin@rootjazz wrote:
Sat Nov 21, 2020 2:56 pm
Screenshot the ACCOUNTS tab and how you create the action. You may be doing it correctly and there is a bug. Or you may be doing it wrong which is why it isn't performing the expected flow. Without seeing the setup, I cannot say which.
A)
Okay.
I tried several different settings (I even think, I tried all possible combinations... :roll: ).
I'll try to reproduce the sequence and document it (with screenshots).

B)
On the other hand: My problem is not really complicated:
I have an account which isn't verified within instadub. I manage to get both
  • SMS codes to my phone number
  • eMail codes to my email inbox
The "only" left problem is how to use these codes to unblock the verify challange.
There shouldn't be tooooooooooo many options in instadub, which are relevant/important to clear the verify challenge?
So there should be some easy recipe what to do? I mean, which options HAVE TO BE right?

C)
martin@rootjazz wrote:
Sat Nov 21, 2020 2:56 pm
Is it because instadub mimicks a PHONE, so instagram thinks there MUST be a phone number and therefore SMS-option available??
No, as I quoted before, instadub "reports" you specified to NOT allow email verification, so it doesn't perform that method
Okay, but isn't it strange that:
The action which TELLS instagram the (correct!) email code fails with the error message that it needs SMS Code????
So I sucessfully (!) initiated an email verify. But WITHIN the action to input this email code, I get the SMS-verfiy-needed-error?
Sounds there's something strange going on to me?

D)
martin@rootjazz wrote:
Sat Nov 21, 2020 2:56 pm
Also I don't understand, why under "Show Optional Advanced" there is the text "Uncheck request code only" all the time. What does it mean?
Check the box and if you have EMAIL and POP settings the program can extract the emailed verification code automatically.
If SMS and you have SMS API details, the program can try and request an SMS to a specific phone number, access that phone inbox and extract the code.
IF you UNCHECK these options, the program will only REQUEST the code to the email / phone, but not try and extract it
Okay, if I understand you correctly:
This text "Uncheck request code only" is nothing more than a static kind of "help text" to describe what the SECOND ROW of check boxes does? (The ones starting with the line "Auto Solve"). So it doesn't change and doesn't tell me that I should do something (that is: uncheck something to solve a problem). Just wasn't 100% sure about whether this text has some deeper meaning and I should take it 'serious' to do something.


User avatar
martin@rootjazz
Site Admin
Posts: 26208
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Problem with 60s delay

Post by martin@rootjazz » Sat Nov 21, 2020 6:16 pm

Faz wrote:
Sat Nov 21, 2020 4:02 pm
ots).

B)
On the other hand: My problem is not really complicated:
I have an account which isn't verified within instadub. I manage to get both
  • SMS codes to my phone number
  • eMail codes to my email inbox
ACCOUNTS tab > LOCAL CODE SUBMIT EMAIL / SMS

Click the link relevant link for SMS / EMAIL , then enter the code you received. That's it. NOTE this ONLY works if instadub requested the code. You cannot request in the browser / manually and submit the code with instadub, instadub must request the keys as it needs to store some IDs to submit the codes

The "only" left problem is how to use these codes to unblock the verify challange.
There shouldn't be tooooooooooo many options in instadub, which are relevant/important to clear the verify challenge?

So there should be some easy recipe what to do? I mean, which options HAVE TO BE right?
I wish it was that simple. Unfortunately the program has to handle all possible work flows IG may show and there is where the complications come in.
This text "Uncheck request code only" is nothing more than a static kind of "help text" to describe what the SECOND ROW of check boxes does? (The ones starting with the line "Auto Solve"). So it doesn't change and doesn't tell me that I should do something (that is: uncheck something to solve a problem).
It is just a note. Check the box to AUTO SOLVE with POP or SMS API or if you leave it unchecked the code is requested and you manually have to get the code and give it to instadub to submit

Faz
Posts: 58
Joined: Thu Mar 15, 2018 12:42 am

Re: Problem with 60s delay

Post by Faz » Sat Nov 21, 2020 8:12 pm

martin@rootjazz wrote:
Sat Nov 21, 2020 6:16 pm
ACCOUNTS tab > LOCAL CODE SUBMIT EMAIL / SMS

Click the link relevant link for SMS / EMAIL , then enter the code you received. That's it. NOTE this ONLY works if instadub requested the code. You cannot request in the browser / manually and submit the code with instadub, instadub must request the keys as it needs to store
Okay, thanks, that's what I'm trying (with all sorts of combinations of options); so there must be something else I overlook. One error I ran into a few times is shown below:
When running the action, it first SUCCEEDS but then - in the very same action - throws A NEW challenge.
I still hope/think there must be some explanation why this would happen.

BTW, it would help to know whether
[ ] Force Login Attempt
[ ] Use Cookies if exist
should be set or not when submitting code. Or whethter these settings are only relevant when running the "Verify:Login" action?

BTW 2: Is there any difference between:
  • "Local Submit Email Cod" via the accounts tab.
  • Going to "Show Optional Advanced", entering the email code there, and clicking "Submit Email Code" here
I guess not?

Code: Select all

Within times start: 2020-11-21 16:13  end: 2020-11-22 08:46 
Starting: 21.11.2020 19:56 
Attempt email code submission: XXX Code: 857063
Extracted code from mail :857063
csrf: aCxFuBuOSIeIO0CWoE3SSSj4pHWkGaiF
Posting to: https://i.instagram.com/challenge/9941065471/5NDkc0GLUq/
Posted: 200
https://i.instagram.com/challenge/9941065471/5NDkc0GLUq/
{"location": "instagram://checkpoint/dismiss", "type": "CHALLENGE_REDIRECTION", "status": "ok"}
got session_id: 9941065471%3AI2ADVuIuE3R35T%3A19
* SUCCESS: posted email code: 857063
* SUCCESS: Verified account: XXX:pw: Proxy: [[[PROXY]]] 
Attempt login to verify code submission successful
Isg login: XXX:pw
Login: XXX:pw Instagram 166.1.0.42.245 Android (23/6.0.1; 480dpi; 1080x1920; Xiaomi; Redmi 4; markw; qcom; en_US)
last_any_login required as last: 02:31:10.9971209 > 60
Forcing login with cookies
Processing login: XXX: Instagram 166.1.0.42.245 Android (23/6.0.1; 480dpi; 1080x1920; Xiaomi; Redmi 4; markw; qcom; en_US)
login: XXX:pw UA: Instagram 166.1.0.42.245 Android (23/6.0.1; 480dpi; 1080x1920; Xiaomi; Redmi 4; markw; qcom; en_US):  Proxy: [[[PROXY]]] 
Generate login data
Generate Signature
Sign Signature
Request failed: (cq6mIMmWW) https://i.instagram.com/api/v1/accounts/login/ 
400
https://i.instagram.com/api/v1/accounts/login/
Instagram error message: challenge_required
* FAILED: Post data: accounts/login: 400 {"message": "challenge_required", "challenge": {"url": "https://i.instagram.com/challenge/9941065471/i0Huql94WJ/", "api_path": "/challenge/9941065471/i0Huql94WJ/", "hide_webview_header": true, "lock": true, "logout": false, "native_flow": true, "flow_render_type": 0, "challenge_context": "{\"step_name\": \"\", \"nonce_code\": \"i0Huql94WJ\", \"user_id\": 9941065471, \"is_stateless\": false}"}, "status": "fail", "error_type": "checkpoint_challenge_required"}
ISG require you verify your account
Checkpoint: https://i.instagram.com/challenge/9941065471/i0Huql94WJ/
ISG asks you to go here: https://i.instagram.com/challenge/9941065471/i0Huql94WJ/ to validate your account again

If you have set your pop details for the account, Instadub can access the email and attempt to verify the account, otherwise it will request the email code and you will have to enter it manually - either by using the browser component to open the checkpoint URL or via the 'Local Submit Email Code' function on the accounts tab.

InstaDUB can open a a session browser for you: double click your account and check the advanced checkbox.

Or if you have SMS API credit (specify on SETTINGS tab) and your account has been marked to allow SMS verification and optionally specified to prefer SMS over email verification (account details form options) Instadub can request the SMS for you to a specified number (for you to receive and submit via ACCOUNTS tab > LOCAL SMS CODE SUBMIT) or handle the whole process automatically (request SMS, receive SMS, submit code)
If a captcha is shown, and you have a 2captcha account and entered your 2captcha API details to the settings tab, Instadub can attempt to solve the captcha for you automatically. More details on 2captcha service: https://rootjazz.com/forum/viewtopic.php?f=15&t=3896
Get checkpoint page: https://i.instagram.com/challenge/9941065471/i0Huql94WJ/
build  api challenge url: https://i.instagram.com/api/v1/challenge/9941065471/i0Huql94WJ/
Found challenge: SelectVerificationMethodForm : {
  "choice": "0",
  "fb_access_token": "None",
  "big_blue_token": "None",
  "google_oauth_token": "true",
  "vetted_device": "None",
  "phone_number": "+1 ***-***-**66",
  "email": "d*******r@w**.de"
}
Email preferred - attempt email first
* EMAIL VERIFICATION not allowed: XXX
Allowed SMS Verification specified
Attempt SMS: 1/3
Request verification SMS
Select SMS option: 
Code already requested, ISG waiting for code submission - will attempt to reset request so can request again
* SUCCESS: challenge process reset
Select SMS option: 
 *** Acc not allowed smsapi: XXX to use SMS API please enable it via accounts form, or accounts list right click menu ***
Submit provided phone number from account: +[[[phonenumber]]]
* SMS code requested: 400
https://i.instagram.com/challenge/9941065471/i0Huql94WJ/
return httpRequestVerification: 400
https://i.instagram.com/challenge/9941065471/i0Huql94WJ/
SMS_FAILED
Attempt SMS: 2/3
Request verification SMS
Select SMS option: 
 *** Acc not allowed smsapi: XXX to use SMS API please enable it via accounts form, or accounts list right click menu ***
Submit provided phone number from account: +[[[phonenumber]]]
* SMS code requested: 400
https://i.instagram.com/challenge/9941065471/i0Huql94WJ/
return httpRequestVerification: 400
https://i.instagram.com/challenge/9941065471/i0Huql94WJ/
SMS_FAILED
Attempt SMS: 3/3
Request verification SMS
Select SMS option: 
 *** Acc not allowed smsapi: XXX to use SMS API please enable it via accounts form, or accounts list right click menu ***
Submit provided phone number from account: +[[[phonenumber]]]
* SMS code requested: 400
https://i.instagram.com/challenge/9941065471/i0Huql94WJ/
return httpRequestVerification: 400
https://i.instagram.com/challenge/9941065471/i0Huql94WJ/
SMS_FAILED
Failed to parse error message: {"message": "challenge_required", "challenge": {"url": "https://i.instagram.com/challenge/9941065471/i0Huql94WJ/", "api_path": "/challenge/9941065471/i0Huql94WJ/", "hide_webview_header": true, "lock": true, "logout": false, "native_flow": true, "flow_render_type": 0, "challenge_context": "{\"step_name\": \"\", \"nonce_code\": \"i0Huql94WJ\", \"user_id\": 9941065471, \"is_stateless\": false}"}, "status": "fail", "error_type": "checkpoint_challenge_required"}
* FAILED Login
* FAILED: Eine Ausnahme vom Typ "LibBotControls.ExFailedLogin" wurde ausgelöst.
Started: 21.11.2020 19:56 
Finished: 21.11.2020 19:57 
ID: b22b31cb-8aee-4112-a45d-794b73548be7
Action ran for: 0hr:0min:44s


User avatar
martin@rootjazz
Site Admin
Posts: 26208
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Problem with 60s delay

Post by martin@rootjazz » Sun Nov 22, 2020 3:43 pm

does the update (maybe you missed it - https://rootjazz.com/instadub/updatetesting.html ) fix the issue of reloading the account specific settings?

User avatar
martin@rootjazz
Site Admin
Posts: 26208
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Problem with 60s delay

Post by martin@rootjazz » Mon Nov 23, 2020 4:31 pm

Faz wrote:
Sat Nov 21, 2020 8:12 pm

When running the action, it first SUCCEEDS but then - in the very same action - throws A NEW challenge.
I still hope/think there must be some explanation why this would happen.
IG may not like something, or it could be a cookie issue, a clean login will rule out cookies:

Goto the ACCOUNTS tab
Check the account
Right click the account and select DELETE COOKIES
Uncheck option to USE EXISTING COOKIES
Check FORCE
click the link VERIFY:LOGIN

BTW, it would help to know whether
[ ] Force Login Attempt
[ ] Use Cookies if exist
should be set or not when submitting code. Or whethter these settings are only relevant when running the "Verify:Login" action?
Not applicable

BTW 2: Is there any difference between:
  • "Local Submit Email Cod" via the accounts tab.
  • Going to "Show Optional Advanced", entering the email code there, and clicking "Submit Email Code" here
I guess not?
Exact same thing
Within times start: 2020-11-21 16:13 end: 2020-11-22 08:46
Starting: 21.11.2020 19:56
Attempt email code submission: XXX Code: 857063
Extracted code from mail :857063
...
* SUCCESS: Verified account: XXX:pw: Proxy: [[[PROXY]]]
Attempt login to verify code submission successful
...
Request failed: (cq6mIMmWW) https://i.instagram.com/api/v1/accounts/login/
400
https://i.instagram.com/api/v1/accounts/login/
Instagram error message: challenge_required
...
* FAILED Login
IG redirect you through the login flow for some reason. Is rare but sometimes they want it perform a couple of terms. If it keeps looping, try a clean login, or you could try a SERVER VERIFY. Is similar process, but you click the SERVER VERIFY link and submit through SERVER SUBMIT CODE links. The action then happens on our server with very high quality IPs which can get past IG looping due to IP



Regards,
Martin

Faz
Posts: 58
Joined: Thu Mar 15, 2018 12:42 am

Re: Problem with 60s delay

Post by Faz » Mon Nov 23, 2020 10:32 pm

Thanks for your help/information.

Status update my login/verify problems: After doing some testing, I think I can say with some confidence, that they were 'instagram problems' (took me some time to do some statistics :D ):

In the past I was able to scrape a lot of users (full CSV + last post info; source: text-file with IDs).
But now instagram seems to
  • block very soon when scraping,
  • flag the account somehow
  • be somehow sensitive if I try to verify/login again through instadub
:?

I found out:
Whenever there's a temporary block (that is: SMS/eMail code verify challange/passwort reset challenge/"I'm not a bot"-having-to-click-on-images challenge/"This-was-me"-challange), it was no problem to clear that temporary block by logging into the account using a NORMAL WEBBROWSER (with a good proxy).

But somehow instagram seems to be able to differentiate between normal webbrowser and instadub lately and bans the accounts when using instadub to verify/login:
Basically all accounts turned to a message like "We look into this. If everything is okay, you get your account back within about 24hours" - and a total ban after some hours/days.

Strange: This ban usually did not happen when:
1 scraping
2 temporary ban
3 clearing the temporary ban using NORMAL WEBBROWSER (so using a normal webbrowser to verify somehow avoided a permanent ban)

So my theory (just guessing): There's a combination of
- flagging account for scraping by instagram
- detecting that there's something fishy when trying to use instadub do verify/clear the block
to get it banned.

This happend even when scraping pretty slowly: Setting working ours (15 h per day) and random pause setting in setting tab: 20 seconds to 60 seconds. As said, in the past it was possible to scrape MUCH MORE MUCH FASTER. But nowadays instagram seems to have implemented some sensitivity to this and flags accounts.

I wonder whether the problem is the MONOTONOUS nature of scraping? Maybe instagram detects that there is bot-behaviour, because the very same action is performed all the time? Maybe it would help to diversify the scraping by throwing in OTHER random actions? But all this doesn't seem to be very practical.

I guess, for the moment, the only workaround would be to start with a lot of accounts and just let them get banned by instagram after using them for a few thousand scrapes. But that would be pretty time consuming and again not really practical.

So IMO the days of being able to scrape many thousand IDs in a few days seem to be over (for now) :?: :? :(

User avatar
martin@rootjazz
Site Admin
Posts: 26208
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Problem with 60s delay

Post by martin@rootjazz » Tue Nov 24, 2020 5:58 pm

Faz wrote:
Mon Nov 23, 2020 10:32 pm

Strange: This ban usually did not happen when:
1 scraping
2 temporary ban
3 clearing the temporary ban using NORMAL WEBBROWSER (so using a normal webbrowser to verify somehow avoided a permanent ban)
and this is ALL you used the accounts for? Scraping - nothing else?
Were what IP? proxy?
How many accounts per IP?
What kind of numbers were you scraping?
Filtering as well? Or just scraping?
How many actions per account (ip) at once?
So my theory (just guessing): There's a combination of
- flagging account for scraping by instagram
- detecting that there's something fishy when trying to use instadub do verify/clear the block
to get it banned.
Always a possibility I won't deny, but I am not hearing of more bans. Certainly not for scraping alone.
As said, in the past it was possible to scrape MUCH MORE MUCH FASTER. But nowadays instagram seems to have implemented some sensitivity to this and flags accounts.
Unfortunately, as you say, things don't stay the same. IG improves detections, reduces limits etc etc.
I guess, for the moment, the only workaround would be to start with a lot of accounts and just let them get banned by instagram after using them for a few thousand scrapes. But that would be pretty time consuming and again not really practical.
a few thousand is a lot if just searching. If scraping then yes, there are orders of magnitude more requests made, but a search returns multiple results, should be fine and I am not hearing of issues here. IF can get back to me with some details / numbers as requested above, we can try and drill into this



Regards,
Martin

Post Reply