Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................X.XXX...X.........................] duration: 676.336149ms FAIL (2025-08-08 00:22:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 474260 (14h23m36.87657683s) [upload=14802]: Got next segment. Inline: false Aug 8 00:03:02.361540620 (14h23m36.876643199s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.361546434 (14h23m36.87664901s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:57.203920880 (14h24m31.719023446s) [seg_pos={0 0}, upload=14802]: Begin upload segment... Aug 8 00:03:57.203988004 (14h24m31.719090583s) [seg_pos={0 0}, upload=14802]: Done begin upload segment. Aug 8 00:03:57.204020830 (14h24m31.719123417s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204041644 (14h24m31.71914423s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204477526 (14h24m31.719580107s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204519100 (14h24m31.719621687s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204895465 (14h24m31.719998052s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204925306 (14h24m31.720027871s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205328523 (14h24m31.720431105s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205769139 (14h24m31.720871721s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207436252 (14h24m31.722538839s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207828410 (14h24m31.722930995s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.208468343 (14h24m31.723570931s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.209378344 (14h24m31.72448093s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.210191405 (14h24m31.72529399s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211289253 (14h24m31.726391837s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211675380 (14h24m31.726777962s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212135467 (14h24m31.727238053s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212506215 (14h24m31.727608796s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212830381 (14h24m31.727932969s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.294294492 (14h24m31.809397074s) [upload=14802]: Next returned nil segment Aug 8 00:03:57.294311485 (14h24m31.809414071s) [upload=14802]: Waiting for error group managing segments... Aug 8 00:03:57.306352633 (14h24m31.821455213s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.307346446 (14h24m31.822449009s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.308700400 (14h24m31.823802978s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.312160131 (14h24m31.827262707s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.320102036 (14h24m31.835204602s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.321164433 (14h24m31.836267014s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.326337358 (14h24m31.841439935s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.352532224 (14h24m31.867634786s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.356261323 (14h24m31.871363886s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.423088525 (14h24m31.938191096s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.471455369 (14h24m31.986557939s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.749384271 (14h24m32.264486838s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.873513358 (14h24m32.388615936s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876598294 (14h24m32.391700872s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876652272 (14h24m32.391754851s) [seg_pos={0 0}, upload=14802]: Segment reached optimal threshold of 14 pieces. Aug 8 00:03:57.877023576 (14h24m32.392126154s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy): protocol: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection Aug 8 00:03:57.877160473 (14h24m32.392263043s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877284905 (14h24m32.392387475s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877449504 (14h24m32.392552073s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ctLLD5skk4t5i3BFYgoTmuZ2aqsX23wDGDMnKriQ7U2KXyPAo): protocol: write tcp 159.223.143.39:47368->104.36.82.3:4794: use of closed network connection; context canceled Aug 8 00:03:57.877469652 (14h24m32.392572221s) [seg_pos={0 0}, upload=14802]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:03:57.877571490 (14h24m32.392674067s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:03:57.877576879 (14h24m32.392679458s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:02.023423762 (14h24m36.538526336s) [upload=14802]: Done uploading segments... Aug 8 00:04:02.023447427 (14h24m36.538550004s) [upload=14802]: Upload finished. err: <nil> Duration: 676.363321ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................XXX.X...XX.XX...............................] duration: 440.900396ms FAIL (2025-08-08 00:22:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130393931 (14h19m30.645496508s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130898433 (14h19m30.646001001s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.131654285 (14h19m30.646756854s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132299167 (14h19m30.647401729s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132815896 (14h19m30.647918459s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.133305230 (14h19m30.648407796s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134064921 (14h19m30.649167503s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134501761 (14h19m30.649604335s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135039838 (14h19m30.650142405s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135450095 (14h19m30.650552665s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135750292 (14h19m30.650852861s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136062113 (14h19m30.651164675s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136413662 (14h19m30.65151623s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136771535 (14h19m30.651874107s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.209688234 (14h19m30.724790803s) [upload=14734]: Next returned nil segment Aug 7 23:58:56.209703014 (14h19m30.72480559s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:58:56.214389537 (14h19m30.729492107s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214445477 (14h19m30.729548054s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214979544 (14h19m30.73008212s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215082462 (14h19m30.730185028s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215826575 (14h19m30.730929141s) [noise=true , address=77.90.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215911546 (14h19m30.731014112s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.216710632 (14h19m30.731813201s) [noise=true , address=77.90.31.18:4807 , piece_id=B3PSWCRJNGDMD6LDDM436N3X75NAA4DEEPLDKL2GTBBJBG3I6ZSQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.217564442 (14h19m30.732667019s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.218100336 (14h19m30.733202913s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.220843127 (14h19m30.735945696s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.221894363 (14h19m30.736996944s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.227829104 (14h19m30.742931668s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326406182 (14h19m30.841508752s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326453057 (14h19m30.841555634s) [seg_pos={0 0}, upload=14734]: Segment reached optimal threshold of 13 pieces. Aug 7 23:58:56.326683605 (14h19m30.841786189s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 23:58:56.326786377 (14h19m30.84188896s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection Aug 7 23:58:56.326888961 (14h19m30.841991543s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: read tcp 159.223.143.39:34618->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:58:56.326913735 (14h19m30.842016315s) [seg_pos={0 0}, upload=14734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:58:56.326998778 (14h19m30.842101358s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:58:56.327004732 (14h19m30.84210731s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:56.375303093 (14h19m30.89040566s) [upload=14734]: Done uploading segments... Aug 7 23:58:56.375321956 (14h19m30.890424523s) [upload=14734]: Upload finished. err: <nil> Aug 7 23:59:26.492160234 (14h20m1.007262811s) [upload=14742]: Starting upload Aug 7 23:59:26.492250238 (14h20m1.007352804s) [upload=14742]: Done starting upload Aug 7 23:59:26.492323252 (14h20m1.00742583s) [upload=14742]: Uploading segments... Aug 7 23:59:26.492605785 (14h20m1.007708349s) [upload=14742]: Got next segment. Inline: false Aug 7 23:59:26.492681402 (14h20m1.007783987s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.492689520 (14h20m1.007792101s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:56.459090933 (14h20m30.974193512s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:56.459165231 (14h20m30.974267796s) [upload=14742]: Done uploading segments... Aug 7 23:59:56.459181138 (14h20m30.974283703s) [upload=14742]: Upload finished. err: context deadline exceeded Aug 7 23:59:56.572495425 (14h20m31.087597994s) [upload=14751]: Starting upload Aug 7 23:59:56.572578962 (14h20m31.087681544s) [upload=14751]: Done starting upload Aug 7 23:59:56.572818459 (14h20m31.087921029s) [upload=14751]: Uploading segments... Aug 7 23:59:56.572953386 (14h20m31.088055953s) [upload=14751]: Got next segment. Inline: false Aug 7 23:59:56.573048904 (14h20m31.088151483s) [upload=14751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:56.573066041 (14h20m31.088168618s) [upload=14751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:26.529103664 (14h21m1.044206252s) [upload=14751]: Next segment err: context deadline exceeded Aug 8 00:00:26.529241226 (14h21m1.044343811s) [upload=14751]: Done uploading segments... Aug 8 00:00:26.529265086 (14h21m1.044367654s) [upload=14751]: Upload finished. err: context deadline exceeded Duration: 440.926836ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.613010686s FAIL (2025-08-08 00:22:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== piece... Aug 8 00:06:47.146014029 (14h27m21.661116615s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146042952 (14h27m21.661145531s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146392229 (14h27m21.661494798s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146489907 (14h27m21.661592493s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146751379 (14h27m21.661853965s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146970578 (14h27m21.662073166s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147102314 (14h27m21.662204899s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147440038 (14h27m21.662542625s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147464022 (14h27m21.662566606s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147806500 (14h27m21.662909085s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148072099 (14h27m21.663174685s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148231380 (14h27m21.663333959s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148503333 (14h27m21.663605919s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151364728 (14h27m21.666467312s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151909137 (14h27m21.667011732s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.152324438 (14h27m21.667427024s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.269249030 (14h27m21.7843516s) [upload=14868]: Next returned nil segment Aug 8 00:06:47.269265635 (14h27m21.784368211s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:06:47.280030107 (14h27m21.795132688s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282243133 (14h27m21.797345711s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282935406 (14h27m21.798037984s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.283797129 (14h27m21.798899704s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.288848212 (14h27m21.803950789s) [noise=true , address=45.91.116.5:4301 , piece_id=GQLYE5Z2Q2SHC2WIEZMTPRGYQ7B4EQZZYJLVC4NQMRENIDZATA5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.292885927 (14h27m21.807988496s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.296224440 (14h27m21.811327016s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.299834425 (14h27m21.81493699s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.305883863 (14h27m21.820986439s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.309497417 (14h27m21.824599987s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.323758805 (14h27m21.838861383s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.326822007 (14h27m21.841924593s) [noise=true , address=104.36.82.45:4938 , piece_id=QI275JMAE4DVWGO5CY3IDTC6WKZCNPPCO3LKE5EK74TGJSBLB2VA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.327668872 (14h27m21.842771449s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339100720 (14h27m21.854203297s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339134259 (14h27m21.854236839s) [seg_pos={0 0}, upload=14868]: Segment reached optimal threshold of 14 pieces. Aug 8 00:06:47.339343619 (14h27m21.854446181s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72): protocol: expected piece hash; context canceled; EOF Aug 8 00:06:47.339560288 (14h27m21.854662868s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mTJmjiHwMJqSkuWKrFnAgJUt1gjQ4wPjfgY7qouWcanaNUy46): protocol: read tcp 159.223.143.39:35278->79.127.205.233:4636: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339701256 (14h27m21.854803837s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: read tcp 159.223.143.39:57032->64.49.13.137:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339837792 (14h27m21.854940379s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk): protocol: read tcp 159.223.143.39:48324->64.49.13.27:4968: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:06:47.339968534 (14h27m21.855071118s) [seg_pos={0 0}, upload=14868]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:06:47.340079739 (14h27m21.855182316s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:06:47.340086599 (14h27m21.855189182s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:47.400977058 (14h27m21.916079641s) [upload=14868]: Done uploading segments... Aug 8 00:06:47.401023653 (14h27m21.916126232s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:07:57.606754320 (14h28m32.121856897s) [upload=14881]: Starting upload Aug 8 00:07:57.606822750 (14h28m32.121925318s) [upload=14881]: Done starting upload Aug 8 00:07:57.607097187 (14h28m32.122199756s) [upload=14881]: Uploading segments... Aug 8 00:07:57.607354625 (14h28m32.122457207s) [upload=14881]: Got next segment. Inline: false Aug 8 00:07:57.607435490 (14h28m32.122538073s) [upload=14881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:57.607441733 (14h28m32.122544313s) [upload=14881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:07:57.622249420 (14h28m32.137352005s) [upload=14881]: Next segment err: metaclient: Too Many Requests Aug 8 00:07:57.622307892 (14h28m32.137410469s) [upload=14881]: Done uploading segments... Aug 8 00:07:57.622322946 (14h28m32.137425515s) [upload=14881]: Upload finished. err: metaclient: Too Many Requests Duration: 1.613037944s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [..................X.XX.X.....X..X...........................] duration: 246.600683ms FAIL (2025-08-08 00:22:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 246.629323ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [...................X..XXX..X...X............................] duration: 236.851428ms FAIL (2025-08-08 00:22:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 236.911767ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.240626793s FAIL (2025-08-08 00:22:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.240677207s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X..........X.............................] duration: 850.248749ms FAIL (2025-08-08 00:22:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 850.318838ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [.......................XXXXXXX.X............................] duration: 255.993324ms FAIL (2025-08-08 00:22:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ne starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:22.831162336 (14h14m57.346264913s) [upload=14661]: Starting upload Aug 7 23:54:22.831234431 (14h14m57.346337008s) [upload=14661]: Done starting upload Aug 7 23:54:22.831306794 (14h14m57.346409362s) [upload=14661]: Uploading segments... Aug 7 23:54:22.831343257 (14h14m57.346445842s) [upload=14661]: Got next segment. Inline: true Aug 7 23:54:22.831347808 (14h14m57.346450375s) [upload=14661]: Waiting for error group managing segments... Aug 7 23:54:22.831399492 (14h14m57.346502071s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:22.831404349 (14h14m57.346506915s) [upload=14661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:22.831407696 (14h14m57.346510265s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:22.799123829 (14h15m57.314226409s) [upload=14661]: Done uploading segments... Aug 7 23:55:22.799154191 (14h15m57.314256772s) [upload=14661]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:22.918485444 (14h15m57.433588022s) [upload=14677]: Starting upload Aug 7 23:55:22.918566621 (14h15m57.433672141s) [upload=14677]: Done starting upload Aug 7 23:55:22.918647605 (14h15m57.433750172s) [upload=14677]: Uploading segments... Aug 7 23:55:22.918705315 (14h15m57.4338079s) [upload=14677]: Got next segment. Inline: true Aug 7 23:55:22.918710403 (14h15m57.43381298s) [upload=14677]: Waiting for error group managing segments... Aug 7 23:55:22.918764176 (14h15m57.433866753s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:22.918769239 (14h15m57.433871815s) [upload=14677]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:22.918772352 (14h15m57.433874928s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:22.891103920 (14h16m57.406206502s) [upload=14677]: Done uploading segments... Aug 7 23:56:22.891142892 (14h16m57.406245471s) [upload=14677]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:23.012512156 (14h16m57.527614733s) [upload=14690]: Starting upload Aug 7 23:56:23.012570683 (14h16m57.527673265s) [upload=14690]: Done starting upload Aug 7 23:56:23.012632890 (14h16m57.527735455s) [upload=14690]: Uploading segments... Aug 7 23:56:23.012675514 (14h16m57.527778093s) [upload=14690]: Got next segment. Inline: true Aug 7 23:56:23.012680516 (14h16m57.527783092s) [upload=14690]: Waiting for error group managing segments... Aug 7 23:56:23.012734720 (14h16m57.527837295s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:23.012739356 (14h16m57.527841923s) [upload=14690]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:23.012742484 (14h16m57.527845059s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:22.981126119 (14h17m57.496228686s) [upload=14690]: Done uploading segments... Aug 7 23:57:22.981169317 (14h17m57.496271889s) [upload=14690]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:23.078169780 (14h17m57.593272348s) [upload=14705]: Starting upload Aug 7 23:57:23.078287503 (14h17m57.593390079s) [upload=14705]: Done starting upload Aug 7 23:57:23.078353530 (14h17m57.593456107s) [upload=14705]: Uploading segments... Aug 7 23:57:23.078394403 (14h17m57.593496992s) [upload=14705]: Got next segment. Inline: true Aug 7 23:57:23.078400888 (14h17m57.593503457s) [upload=14705]: Waiting for error group managing segments... Aug 7 23:57:23.078452630 (14h17m57.593555201s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:23.078457270 (14h17m57.593559838s) [upload=14705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:23.078460483 (14h17m57.593563048s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:23.049396934 (14h18m57.564499518s) [upload=14705]: Done uploading segments... Aug 7 23:58:23.049418059 (14h18m57.564520634s) [upload=14705]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.169720285 (14h18m57.684822852s) [upload=14722]: Starting upload Aug 7 23:58:23.169779212 (14h18m57.684881788s) [upload=14722]: Done starting upload Aug 7 23:58:23.169840834 (14h18m57.684943401s) [upload=14722]: Uploading segments... Aug 7 23:58:23.169880437 (14h18m57.684983021s) [upload=14722]: Got next segment. Inline: true Aug 7 23:58:23.169885772 (14h18m57.684988338s) [upload=14722]: Waiting for error group managing segments... Aug 7 23:58:23.169927728 (14h18m57.685030303s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.169933919 (14h18m57.685036498s) [upload=14722]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.169938117 (14h18m57.685040704s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:23.138140642 (14h19m57.653243209s) [upload=14722]: Done uploading segments... Aug 7 23:59:23.138159469 (14h19m57.653262035s) [upload=14722]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:35.502179618 (14h20m10.017282184s) [upload=14745]: Starting upload Aug 7 23:59:35.502246881 (14h20m10.017349449s) [upload=14745]: Done starting upload Aug 7 23:59:35.502308502 (14h20m10.017411071s) [upload=14745]: Uploading segments... Aug 7 23:59:35.502347394 (14h20m10.017449979s) [upload=14745]: Got next segment. Inline: true Aug 7 23:59:35.502351750 (14h20m10.017454327s) [upload=14745]: Waiting for error group managing segments... Aug 7 23:59:35.502390443 (14h20m10.017493009s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:35.502394821 (14h20m10.017497398s) [upload=14745]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:35.502398119 (14h20m10.017500695s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.472447782 (14h21m9.987550364s) [upload=14745]: Done uploading segments... Aug 8 00:00:35.472473966 (14h21m9.987576558s) [upload=14745]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:35.593243192 (14h21m10.10834576s) [upload=14760]: Starting upload Aug 8 00:00:35.593395835 (14h21m10.108498415s) [upload=14760]: Done starting upload Aug 8 00:00:35.593497926 (14h21m10.10860049s) [upload=14760]: Uploading segments... Aug 8 00:00:35.593659481 (14h21m10.108762045s) [upload=14760]: Got next segment. Inline: true Aug 8 00:00:35.593667686 (14h21m10.108770273s) [upload=14760]: Waiting for error group managing segments... Aug 8 00:00:35.593750345 (14h21m10.10885293s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.593773930 (14h21m10.108876508s) [upload=14760]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.593782409 (14h21m10.108884991s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.662793141 (14h21m10.177895709s) [upload=14760]: Done uploading segments... Aug 8 00:00:35.662815087 (14h21m10.177917658s) [upload=14760]: Upload finished. err: <nil> Aug 8 00:01:23.648690375 (14h21m58.163792959s) [upload=14775]: Starting upload Aug 8 00:01:23.648799550 (14h21m58.163902126s) [upload=14775]: Done starting upload Aug 8 00:01:23.648928369 (14h21m58.164030934s) [upload=14775]: Uploading segments... Aug 8 00:01:23.648986999 (14h21m58.164089578s) [upload=14775]: Got next segment. Inline: true Aug 8 00:01:23.648992059 (14h21m58.164094634s) [upload=14775]: Waiting for error group managing segments... Aug 8 00:01:23.649037148 (14h21m58.164139714s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:23.649041722 (14h21m58.164144298s) [upload=14775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:23.649044950 (14h21m58.164147527s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:23.618855090 (14h22m58.133957671s) [upload=14775]: Done uploading segments... Aug 8 00:02:23.618900564 (14h22m58.134003142s) [upload=14775]: Upload finished. err: metaclient: context deadline exceeded Duration: 256.113991ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................X.XXX...X........................] duration: 671.502415ms FAIL (2025-08-08 00:21:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 474260 (14h23m36.87657683s) [upload=14802]: Got next segment. Inline: false Aug 8 00:03:02.361540620 (14h23m36.876643199s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.361546434 (14h23m36.87664901s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:57.203920880 (14h24m31.719023446s) [seg_pos={0 0}, upload=14802]: Begin upload segment... Aug 8 00:03:57.203988004 (14h24m31.719090583s) [seg_pos={0 0}, upload=14802]: Done begin upload segment. Aug 8 00:03:57.204020830 (14h24m31.719123417s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204041644 (14h24m31.71914423s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204477526 (14h24m31.719580107s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204519100 (14h24m31.719621687s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204895465 (14h24m31.719998052s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204925306 (14h24m31.720027871s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205328523 (14h24m31.720431105s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205769139 (14h24m31.720871721s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207436252 (14h24m31.722538839s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207828410 (14h24m31.722930995s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.208468343 (14h24m31.723570931s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.209378344 (14h24m31.72448093s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.210191405 (14h24m31.72529399s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211289253 (14h24m31.726391837s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211675380 (14h24m31.726777962s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212135467 (14h24m31.727238053s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212506215 (14h24m31.727608796s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212830381 (14h24m31.727932969s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.294294492 (14h24m31.809397074s) [upload=14802]: Next returned nil segment Aug 8 00:03:57.294311485 (14h24m31.809414071s) [upload=14802]: Waiting for error group managing segments... Aug 8 00:03:57.306352633 (14h24m31.821455213s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.307346446 (14h24m31.822449009s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.308700400 (14h24m31.823802978s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.312160131 (14h24m31.827262707s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.320102036 (14h24m31.835204602s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.321164433 (14h24m31.836267014s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.326337358 (14h24m31.841439935s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.352532224 (14h24m31.867634786s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.356261323 (14h24m31.871363886s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.423088525 (14h24m31.938191096s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.471455369 (14h24m31.986557939s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.749384271 (14h24m32.264486838s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.873513358 (14h24m32.388615936s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876598294 (14h24m32.391700872s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876652272 (14h24m32.391754851s) [seg_pos={0 0}, upload=14802]: Segment reached optimal threshold of 14 pieces. Aug 8 00:03:57.877023576 (14h24m32.392126154s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy): protocol: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection Aug 8 00:03:57.877160473 (14h24m32.392263043s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877284905 (14h24m32.392387475s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877449504 (14h24m32.392552073s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ctLLD5skk4t5i3BFYgoTmuZ2aqsX23wDGDMnKriQ7U2KXyPAo): protocol: write tcp 159.223.143.39:47368->104.36.82.3:4794: use of closed network connection; context canceled Aug 8 00:03:57.877469652 (14h24m32.392572221s) [seg_pos={0 0}, upload=14802]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:03:57.877571490 (14h24m32.392674067s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:03:57.877576879 (14h24m32.392679458s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:02.023423762 (14h24m36.538526336s) [upload=14802]: Done uploading segments... Aug 8 00:04:02.023447427 (14h24m36.538550004s) [upload=14802]: Upload finished. err: <nil> Duration: 671.5342ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 822.48691ms FAIL (2025-08-08 00:21:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== piece... Aug 8 00:06:47.146014029 (14h27m21.661116615s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146042952 (14h27m21.661145531s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146392229 (14h27m21.661494798s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146489907 (14h27m21.661592493s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146751379 (14h27m21.661853965s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146970578 (14h27m21.662073166s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147102314 (14h27m21.662204899s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147440038 (14h27m21.662542625s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147464022 (14h27m21.662566606s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147806500 (14h27m21.662909085s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148072099 (14h27m21.663174685s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148231380 (14h27m21.663333959s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148503333 (14h27m21.663605919s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151364728 (14h27m21.666467312s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151909137 (14h27m21.667011732s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.152324438 (14h27m21.667427024s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.269249030 (14h27m21.7843516s) [upload=14868]: Next returned nil segment Aug 8 00:06:47.269265635 (14h27m21.784368211s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:06:47.280030107 (14h27m21.795132688s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282243133 (14h27m21.797345711s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282935406 (14h27m21.798037984s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.283797129 (14h27m21.798899704s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.288848212 (14h27m21.803950789s) [noise=true , address=45.91.116.5:4301 , piece_id=GQLYE5Z2Q2SHC2WIEZMTPRGYQ7B4EQZZYJLVC4NQMRENIDZATA5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.292885927 (14h27m21.807988496s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.296224440 (14h27m21.811327016s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.299834425 (14h27m21.81493699s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.305883863 (14h27m21.820986439s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.309497417 (14h27m21.824599987s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.323758805 (14h27m21.838861383s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.326822007 (14h27m21.841924593s) [noise=true , address=104.36.82.45:4938 , piece_id=QI275JMAE4DVWGO5CY3IDTC6WKZCNPPCO3LKE5EK74TGJSBLB2VA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.327668872 (14h27m21.842771449s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339100720 (14h27m21.854203297s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339134259 (14h27m21.854236839s) [seg_pos={0 0}, upload=14868]: Segment reached optimal threshold of 14 pieces. Aug 8 00:06:47.339343619 (14h27m21.854446181s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72): protocol: expected piece hash; context canceled; EOF Aug 8 00:06:47.339560288 (14h27m21.854662868s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mTJmjiHwMJqSkuWKrFnAgJUt1gjQ4wPjfgY7qouWcanaNUy46): protocol: read tcp 159.223.143.39:35278->79.127.205.233:4636: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339701256 (14h27m21.854803837s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: read tcp 159.223.143.39:57032->64.49.13.137:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339837792 (14h27m21.854940379s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk): protocol: read tcp 159.223.143.39:48324->64.49.13.27:4968: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:06:47.339968534 (14h27m21.855071118s) [seg_pos={0 0}, upload=14868]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:06:47.340079739 (14h27m21.855182316s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:06:47.340086599 (14h27m21.855189182s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:47.400977058 (14h27m21.916079641s) [upload=14868]: Done uploading segments... Aug 8 00:06:47.401023653 (14h27m21.916126232s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:07:57.606754320 (14h28m32.121856897s) [upload=14881]: Starting upload Aug 8 00:07:57.606822750 (14h28m32.121925318s) [upload=14881]: Done starting upload Aug 8 00:07:57.607097187 (14h28m32.122199756s) [upload=14881]: Uploading segments... Aug 8 00:07:57.607354625 (14h28m32.122457207s) [upload=14881]: Got next segment. Inline: false Aug 8 00:07:57.607435490 (14h28m32.122538073s) [upload=14881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:57.607441733 (14h28m32.122544313s) [upload=14881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:07:57.622249420 (14h28m32.137352005s) [upload=14881]: Next segment err: metaclient: Too Many Requests Aug 8 00:07:57.622307892 (14h28m32.137410469s) [upload=14881]: Done uploading segments... Aug 8 00:07:57.622322946 (14h28m32.137425515s) [upload=14881]: Upload finished. err: metaclient: Too Many Requests Duration: 822.529346ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 424.996866ms FAIL (2025-08-08 00:21:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== g segments... Aug 8 00:01:46.657245436 (14h22m21.172348015s) [upload=14780]: Upload finished. err: <nil> Aug 8 00:02:30.275266890 (14h23m4.790369467s) [upload=14795]: Starting upload Aug 8 00:02:30.275331708 (14h23m4.790434285s) [upload=14795]: Done starting upload Aug 8 00:02:30.275419113 (14h23m4.790521681s) [upload=14795]: Uploading segments... Aug 8 00:02:30.275464303 (14h23m4.790566902s) [upload=14795]: Got next segment. Inline: true Aug 8 00:02:30.275470409 (14h23m4.790572986s) [upload=14795]: Waiting for error group managing segments... Aug 8 00:02:30.275541468 (14h23m4.790644064s) [upload=14795]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.275548435 (14h23m4.790651025s) [upload=14795]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:30.275553112 (14h23m4.790655694s) [upload=14795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.437129275 (14h23m4.952231852s) [upload=14795]: Done uploading segments... Aug 8 00:02:30.437147837 (14h23m4.952250404s) [upload=14795]: Upload finished. err: <nil> Aug 8 00:03:02.216029168 (14h23m36.731131753s) [upload=14801]: Starting upload Aug 8 00:03:02.216131444 (14h23m36.731234024s) [upload=14801]: Done starting upload Aug 8 00:03:02.216223262 (14h23m36.731325836s) [upload=14801]: Uploading segments... Aug 8 00:03:02.216305099 (14h23m36.731407689s) [upload=14801]: Got next segment. Inline: true Aug 8 00:03:02.216311582 (14h23m36.731414165s) [upload=14801]: Waiting for error group managing segments... Aug 8 00:03:02.216381029 (14h23m36.731483612s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.216386785 (14h23m36.731489374s) [upload=14801]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:02.216390618 (14h23m36.731493197s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:02.283802345 (14h23m36.798904923s) [upload=14801]: Done uploading segments... Aug 8 00:03:02.283823305 (14h23m36.798925889s) [upload=14801]: Upload finished. err: <nil> Aug 8 00:03:30.509760483 (14h24m5.024863047s) [upload=14812]: Starting upload Aug 8 00:03:30.509815323 (14h24m5.024917908s) [upload=14812]: Done starting upload Aug 8 00:03:30.509926692 (14h24m5.02502927s) [upload=14812]: Uploading segments... Aug 8 00:03:30.509996173 (14h24m5.025098754s) [upload=14812]: Got next segment. Inline: true Aug 8 00:03:30.510002796 (14h24m5.025105377s) [upload=14812]: Waiting for error group managing segments... Aug 8 00:03:30.510068440 (14h24m5.025171017s) [upload=14812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:30.510074098 (14h24m5.025176685s) [upload=14812]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:30.510077690 (14h24m5.025180273s) [upload=14812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:30.698239096 (14h24m5.213341663s) [upload=14812]: Done uploading segments... Aug 8 00:03:30.698256740 (14h24m5.213359308s) [upload=14812]: Upload finished. err: <nil> Aug 8 00:04:41.065788390 (14h25m15.580890975s) [upload=14831]: Starting upload Aug 8 00:04:41.065849488 (14h25m15.580952064s) [upload=14831]: Done starting upload Aug 8 00:04:41.065914279 (14h25m15.581016856s) [upload=14831]: Uploading segments... Aug 8 00:04:41.065955528 (14h25m15.581058114s) [upload=14831]: Got next segment. Inline: true Aug 8 00:04:41.065961233 (14h25m15.581063813s) [upload=14831]: Waiting for error group managing segments... Aug 8 00:04:41.066010814 (14h25m15.581113395s) [upload=14831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:04:41.066015217 (14h25m15.581117797s) [upload=14831]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:04:41.066019044 (14h25m15.581121626s) [upload=14831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:41.143560517 (14h25m15.658663093s) [upload=14831]: Done uploading segments... Aug 8 00:04:41.143578273 (14h25m15.658680839s) [upload=14831]: Upload finished. err: <nil> Aug 8 00:04:57.174663208 (14h25m31.689765785s) [upload=14836]: Starting upload Aug 8 00:04:57.174738504 (14h25m31.68984108s) [upload=14836]: Done starting upload Aug 8 00:04:57.174818503 (14h25m31.689921072s) [upload=14836]: Uploading segments... Aug 8 00:04:57.174883379 (14h25m31.689985957s) [upload=14836]: Got next segment. Inline: true Aug 8 00:04:57.174888780 (14h25m31.689991347s) [upload=14836]: Waiting for error group managing segments... Aug 8 00:04:57.174955675 (14h25m31.690058262s) [upload=14836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:04:57.174962095 (14h25m31.69006468s) [upload=14836]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:04:57.174966595 (14h25m31.690069178s) [upload=14836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:57.317214071 (14h25m31.832316638s) [upload=14836]: Done uploading segments... Aug 8 00:04:57.317241163 (14h25m31.832343733s) [upload=14836]: Upload finished. err: <nil> Aug 8 00:06:06.741579891 (14h26m41.256682461s) [upload=14855]: Starting upload Aug 8 00:06:06.741694208 (14h26m41.256796791s) [upload=14855]: Done starting upload Aug 8 00:06:06.741776231 (14h26m41.256878801s) [upload=14855]: Uploading segments... Aug 8 00:06:06.741863808 (14h26m41.256966377s) [upload=14855]: Got next segment. Inline: true Aug 8 00:06:06.741870254 (14h26m41.25697283s) [upload=14855]: Waiting for error group managing segments... Aug 8 00:06:06.741939630 (14h26m41.257042211s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:06:06.741944620 (14h26m41.257047201s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:06:06.741950358 (14h26m41.257052937s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:07.276784579 (14h26m41.791887173s) [upload=14855]: Done uploading segments... Aug 8 00:06:07.276805845 (14h26m41.79190841s) [upload=14855]: Upload finished. err: <nil> Aug 8 00:06:43.194994307 (14h27m17.710096878s) [upload=14865]: Starting upload Aug 8 00:06:43.195063177 (14h27m17.710165753s) [upload=14865]: Done starting upload Aug 8 00:06:43.195131851 (14h27m17.710234419s) [upload=14865]: Uploading segments... Aug 8 00:06:43.195200076 (14h27m17.710302661s) [upload=14865]: Got next segment. Inline: true Aug 8 00:06:43.195205915 (14h27m17.710308491s) [upload=14865]: Waiting for error group managing segments... Aug 8 00:06:43.195271114 (14h27m17.710373692s) [upload=14865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:06:43.195276311 (14h27m17.710378889s) [upload=14865]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:06:43.195279806 (14h27m17.710382383s) [upload=14865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:43.430698626 (14h27m17.945801203s) [upload=14865]: Done uploading segments... Aug 8 00:06:43.430715582 (14h27m17.945818158s) [upload=14865]: Upload finished. err: <nil> Aug 8 00:07:00.838748208 (14h27m35.353850791s) [upload=14870]: Starting upload Aug 8 00:07:00.838831814 (14h27m35.353934397s) [upload=14870]: Done starting upload Aug 8 00:07:00.838944876 (14h27m35.354047439s) [upload=14870]: Uploading segments... Aug 8 00:07:00.839014925 (14h27m35.354117501s) [upload=14870]: Got next segment. Inline: true Aug 8 00:07:00.839045878 (14h27m35.354148457s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:07:00.839120971 (14h27m35.354223549s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:00.839126346 (14h27m35.354228923s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:00.839129942 (14h27m35.354232518s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:00.932912415 (14h27m35.448014989s) [upload=14870]: Done uploading segments... Aug 8 00:07:00.932927726 (14h27m35.448030289s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:08:14.384482900 (14h28m48.899585476s) [upload=14888]: Starting upload Aug 8 00:08:14.384593866 (14h28m48.899696446s) [upload=14888]: Done starting upload Aug 8 00:08:14.384673686 (14h28m48.899776254s) [upload=14888]: Uploading segments... Aug 8 00:08:14.384718832 (14h28m48.899821412s) [upload=14888]: Got next segment. Inline: true Aug 8 00:08:14.384723835 (14h28m48.899826412s) [upload=14888]: Waiting for error group managing segments... Aug 8 00:08:14.384763544 (14h28m48.89986612s) [upload=14888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:14.384768731 (14h28m48.899871307s) [upload=14888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:14.384771994 (14h28m48.89987457s) [upload=14888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:14.710769235 (14h28m49.225871818s) [upload=14888]: Done uploading segments... Aug 8 00:08:14.710803190 (14h28m49.225905767s) [upload=14888]: Upload finished. err: <nil> Aug 8 00:09:18.634609803 (14h29m53.149712376s) [upload=14909]: Starting upload Aug 8 00:09:18.634680614 (14h29m53.149783189s) [upload=14909]: Done starting upload Aug 8 00:09:18.634765976 (14h29m53.149868543s) [upload=14909]: Uploading segments... Aug 8 00:09:18.634815721 (14h29m53.1499183s) [upload=14909]: Got next segment. Inline: true Aug 8 00:09:18.634820719 (14h29m53.149923286s) [upload=14909]: Waiting for error group managing segments... Aug 8 00:09:18.634863856 (14h29m53.149966435s) [upload=14909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:09:18.634868286 (14h29m53.149970863s) [upload=14909]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:09:18.634872291 (14h29m53.14997487s) [upload=14909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:09:18.642683461 (14h29m53.157786047s) [upload=14909]: Done uploading segments... Aug 8 00:09:18.642703785 (14h29m53.157806368s) [upload=14909]: Upload finished. err: metaclient: Too Many Requests Duration: 425.022045ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................XXX.X...XX.XX..............................] duration: 484.694947ms FAIL (2025-08-08 00:21:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130393931 (14h19m30.645496508s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130898433 (14h19m30.646001001s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.131654285 (14h19m30.646756854s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132299167 (14h19m30.647401729s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132815896 (14h19m30.647918459s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.133305230 (14h19m30.648407796s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134064921 (14h19m30.649167503s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134501761 (14h19m30.649604335s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135039838 (14h19m30.650142405s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135450095 (14h19m30.650552665s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135750292 (14h19m30.650852861s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136062113 (14h19m30.651164675s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136413662 (14h19m30.65151623s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136771535 (14h19m30.651874107s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.209688234 (14h19m30.724790803s) [upload=14734]: Next returned nil segment Aug 7 23:58:56.209703014 (14h19m30.72480559s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:58:56.214389537 (14h19m30.729492107s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214445477 (14h19m30.729548054s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214979544 (14h19m30.73008212s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215082462 (14h19m30.730185028s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215826575 (14h19m30.730929141s) [noise=true , address=77.90.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215911546 (14h19m30.731014112s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.216710632 (14h19m30.731813201s) [noise=true , address=77.90.31.18:4807 , piece_id=B3PSWCRJNGDMD6LDDM436N3X75NAA4DEEPLDKL2GTBBJBG3I6ZSQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.217564442 (14h19m30.732667019s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.218100336 (14h19m30.733202913s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.220843127 (14h19m30.735945696s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.221894363 (14h19m30.736996944s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.227829104 (14h19m30.742931668s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326406182 (14h19m30.841508752s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326453057 (14h19m30.841555634s) [seg_pos={0 0}, upload=14734]: Segment reached optimal threshold of 13 pieces. Aug 7 23:58:56.326683605 (14h19m30.841786189s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 23:58:56.326786377 (14h19m30.84188896s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection Aug 7 23:58:56.326888961 (14h19m30.841991543s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: read tcp 159.223.143.39:34618->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:58:56.326913735 (14h19m30.842016315s) [seg_pos={0 0}, upload=14734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:58:56.326998778 (14h19m30.842101358s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:58:56.327004732 (14h19m30.84210731s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:56.375303093 (14h19m30.89040566s) [upload=14734]: Done uploading segments... Aug 7 23:58:56.375321956 (14h19m30.890424523s) [upload=14734]: Upload finished. err: <nil> Aug 7 23:59:26.492160234 (14h20m1.007262811s) [upload=14742]: Starting upload Aug 7 23:59:26.492250238 (14h20m1.007352804s) [upload=14742]: Done starting upload Aug 7 23:59:26.492323252 (14h20m1.00742583s) [upload=14742]: Uploading segments... Aug 7 23:59:26.492605785 (14h20m1.007708349s) [upload=14742]: Got next segment. Inline: false Aug 7 23:59:26.492681402 (14h20m1.007783987s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.492689520 (14h20m1.007792101s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:56.459090933 (14h20m30.974193512s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:56.459165231 (14h20m30.974267796s) [upload=14742]: Done uploading segments... Aug 7 23:59:56.459181138 (14h20m30.974283703s) [upload=14742]: Upload finished. err: context deadline exceeded Aug 7 23:59:56.572495425 (14h20m31.087597994s) [upload=14751]: Starting upload Aug 7 23:59:56.572578962 (14h20m31.087681544s) [upload=14751]: Done starting upload Aug 7 23:59:56.572818459 (14h20m31.087921029s) [upload=14751]: Uploading segments... Aug 7 23:59:56.572953386 (14h20m31.088055953s) [upload=14751]: Got next segment. Inline: false Aug 7 23:59:56.573048904 (14h20m31.088151483s) [upload=14751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:56.573066041 (14h20m31.088168618s) [upload=14751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:26.529103664 (14h21m1.044206252s) [upload=14751]: Next segment err: context deadline exceeded Aug 8 00:00:26.529241226 (14h21m1.044343811s) [upload=14751]: Done uploading segments... Aug 8 00:00:26.529265086 (14h21m1.044367654s) [upload=14751]: Upload finished. err: context deadline exceeded Duration: 484.727234ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................XX..XX..............................] duration: 667.197965ms FAIL (2025-08-08 00:21:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 667.222848ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................X..X....XX...X..........................] duration: 252.606128ms FAIL (2025-08-08 00:21:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 252.641956ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [........................XXXXXXX.X...........................] duration: 232.056913ms FAIL (2025-08-08 00:21:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ne starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:22.831162336 (14h14m57.346264913s) [upload=14661]: Starting upload Aug 7 23:54:22.831234431 (14h14m57.346337008s) [upload=14661]: Done starting upload Aug 7 23:54:22.831306794 (14h14m57.346409362s) [upload=14661]: Uploading segments... Aug 7 23:54:22.831343257 (14h14m57.346445842s) [upload=14661]: Got next segment. Inline: true Aug 7 23:54:22.831347808 (14h14m57.346450375s) [upload=14661]: Waiting for error group managing segments... Aug 7 23:54:22.831399492 (14h14m57.346502071s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:22.831404349 (14h14m57.346506915s) [upload=14661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:22.831407696 (14h14m57.346510265s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:22.799123829 (14h15m57.314226409s) [upload=14661]: Done uploading segments... Aug 7 23:55:22.799154191 (14h15m57.314256772s) [upload=14661]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:22.918485444 (14h15m57.433588022s) [upload=14677]: Starting upload Aug 7 23:55:22.918566621 (14h15m57.433672141s) [upload=14677]: Done starting upload Aug 7 23:55:22.918647605 (14h15m57.433750172s) [upload=14677]: Uploading segments... Aug 7 23:55:22.918705315 (14h15m57.4338079s) [upload=14677]: Got next segment. Inline: true Aug 7 23:55:22.918710403 (14h15m57.43381298s) [upload=14677]: Waiting for error group managing segments... Aug 7 23:55:22.918764176 (14h15m57.433866753s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:22.918769239 (14h15m57.433871815s) [upload=14677]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:22.918772352 (14h15m57.433874928s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:22.891103920 (14h16m57.406206502s) [upload=14677]: Done uploading segments... Aug 7 23:56:22.891142892 (14h16m57.406245471s) [upload=14677]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:23.012512156 (14h16m57.527614733s) [upload=14690]: Starting upload Aug 7 23:56:23.012570683 (14h16m57.527673265s) [upload=14690]: Done starting upload Aug 7 23:56:23.012632890 (14h16m57.527735455s) [upload=14690]: Uploading segments... Aug 7 23:56:23.012675514 (14h16m57.527778093s) [upload=14690]: Got next segment. Inline: true Aug 7 23:56:23.012680516 (14h16m57.527783092s) [upload=14690]: Waiting for error group managing segments... Aug 7 23:56:23.012734720 (14h16m57.527837295s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:23.012739356 (14h16m57.527841923s) [upload=14690]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:23.012742484 (14h16m57.527845059s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:22.981126119 (14h17m57.496228686s) [upload=14690]: Done uploading segments... Aug 7 23:57:22.981169317 (14h17m57.496271889s) [upload=14690]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:23.078169780 (14h17m57.593272348s) [upload=14705]: Starting upload Aug 7 23:57:23.078287503 (14h17m57.593390079s) [upload=14705]: Done starting upload Aug 7 23:57:23.078353530 (14h17m57.593456107s) [upload=14705]: Uploading segments... Aug 7 23:57:23.078394403 (14h17m57.593496992s) [upload=14705]: Got next segment. Inline: true Aug 7 23:57:23.078400888 (14h17m57.593503457s) [upload=14705]: Waiting for error group managing segments... Aug 7 23:57:23.078452630 (14h17m57.593555201s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:23.078457270 (14h17m57.593559838s) [upload=14705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:23.078460483 (14h17m57.593563048s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:23.049396934 (14h18m57.564499518s) [upload=14705]: Done uploading segments... Aug 7 23:58:23.049418059 (14h18m57.564520634s) [upload=14705]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.169720285 (14h18m57.684822852s) [upload=14722]: Starting upload Aug 7 23:58:23.169779212 (14h18m57.684881788s) [upload=14722]: Done starting upload Aug 7 23:58:23.169840834 (14h18m57.684943401s) [upload=14722]: Uploading segments... Aug 7 23:58:23.169880437 (14h18m57.684983021s) [upload=14722]: Got next segment. Inline: true Aug 7 23:58:23.169885772 (14h18m57.684988338s) [upload=14722]: Waiting for error group managing segments... Aug 7 23:58:23.169927728 (14h18m57.685030303s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.169933919 (14h18m57.685036498s) [upload=14722]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.169938117 (14h18m57.685040704s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:23.138140642 (14h19m57.653243209s) [upload=14722]: Done uploading segments... Aug 7 23:59:23.138159469 (14h19m57.653262035s) [upload=14722]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:35.502179618 (14h20m10.017282184s) [upload=14745]: Starting upload Aug 7 23:59:35.502246881 (14h20m10.017349449s) [upload=14745]: Done starting upload Aug 7 23:59:35.502308502 (14h20m10.017411071s) [upload=14745]: Uploading segments... Aug 7 23:59:35.502347394 (14h20m10.017449979s) [upload=14745]: Got next segment. Inline: true Aug 7 23:59:35.502351750 (14h20m10.017454327s) [upload=14745]: Waiting for error group managing segments... Aug 7 23:59:35.502390443 (14h20m10.017493009s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:35.502394821 (14h20m10.017497398s) [upload=14745]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:35.502398119 (14h20m10.017500695s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.472447782 (14h21m9.987550364s) [upload=14745]: Done uploading segments... Aug 8 00:00:35.472473966 (14h21m9.987576558s) [upload=14745]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:35.593243192 (14h21m10.10834576s) [upload=14760]: Starting upload Aug 8 00:00:35.593395835 (14h21m10.108498415s) [upload=14760]: Done starting upload Aug 8 00:00:35.593497926 (14h21m10.10860049s) [upload=14760]: Uploading segments... Aug 8 00:00:35.593659481 (14h21m10.108762045s) [upload=14760]: Got next segment. Inline: true Aug 8 00:00:35.593667686 (14h21m10.108770273s) [upload=14760]: Waiting for error group managing segments... Aug 8 00:00:35.593750345 (14h21m10.10885293s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.593773930 (14h21m10.108876508s) [upload=14760]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.593782409 (14h21m10.108884991s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.662793141 (14h21m10.177895709s) [upload=14760]: Done uploading segments... Aug 8 00:00:35.662815087 (14h21m10.177917658s) [upload=14760]: Upload finished. err: <nil> Aug 8 00:01:23.648690375 (14h21m58.163792959s) [upload=14775]: Starting upload Aug 8 00:01:23.648799550 (14h21m58.163902126s) [upload=14775]: Done starting upload Aug 8 00:01:23.648928369 (14h21m58.164030934s) [upload=14775]: Uploading segments... Aug 8 00:01:23.648986999 (14h21m58.164089578s) [upload=14775]: Got next segment. Inline: true Aug 8 00:01:23.648992059 (14h21m58.164094634s) [upload=14775]: Waiting for error group managing segments... Aug 8 00:01:23.649037148 (14h21m58.164139714s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:23.649041722 (14h21m58.164144298s) [upload=14775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:23.649044950 (14h21m58.164147527s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:23.618855090 (14h22m58.133957671s) [upload=14775]: Done uploading segments... Aug 8 00:02:23.618900564 (14h22m58.134003142s) [upload=14775]: Upload finished. err: metaclient: context deadline exceeded Duration: 232.100543ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................XXX.X...XX.XX.............................] duration: 471.973598ms FAIL (2025-08-08 00:21:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130393931 (14h19m30.645496508s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130898433 (14h19m30.646001001s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.131654285 (14h19m30.646756854s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132299167 (14h19m30.647401729s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132815896 (14h19m30.647918459s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.133305230 (14h19m30.648407796s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134064921 (14h19m30.649167503s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134501761 (14h19m30.649604335s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135039838 (14h19m30.650142405s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135450095 (14h19m30.650552665s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135750292 (14h19m30.650852861s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136062113 (14h19m30.651164675s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136413662 (14h19m30.65151623s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136771535 (14h19m30.651874107s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.209688234 (14h19m30.724790803s) [upload=14734]: Next returned nil segment Aug 7 23:58:56.209703014 (14h19m30.72480559s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:58:56.214389537 (14h19m30.729492107s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214445477 (14h19m30.729548054s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214979544 (14h19m30.73008212s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215082462 (14h19m30.730185028s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215826575 (14h19m30.730929141s) [noise=true , address=77.90.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215911546 (14h19m30.731014112s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.216710632 (14h19m30.731813201s) [noise=true , address=77.90.31.18:4807 , piece_id=B3PSWCRJNGDMD6LDDM436N3X75NAA4DEEPLDKL2GTBBJBG3I6ZSQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.217564442 (14h19m30.732667019s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.218100336 (14h19m30.733202913s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.220843127 (14h19m30.735945696s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.221894363 (14h19m30.736996944s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.227829104 (14h19m30.742931668s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326406182 (14h19m30.841508752s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326453057 (14h19m30.841555634s) [seg_pos={0 0}, upload=14734]: Segment reached optimal threshold of 13 pieces. Aug 7 23:58:56.326683605 (14h19m30.841786189s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 23:58:56.326786377 (14h19m30.84188896s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection Aug 7 23:58:56.326888961 (14h19m30.841991543s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: read tcp 159.223.143.39:34618->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:58:56.326913735 (14h19m30.842016315s) [seg_pos={0 0}, upload=14734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:58:56.326998778 (14h19m30.842101358s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:58:56.327004732 (14h19m30.84210731s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:56.375303093 (14h19m30.89040566s) [upload=14734]: Done uploading segments... Aug 7 23:58:56.375321956 (14h19m30.890424523s) [upload=14734]: Upload finished. err: <nil> Aug 7 23:59:26.492160234 (14h20m1.007262811s) [upload=14742]: Starting upload Aug 7 23:59:26.492250238 (14h20m1.007352804s) [upload=14742]: Done starting upload Aug 7 23:59:26.492323252 (14h20m1.00742583s) [upload=14742]: Uploading segments... Aug 7 23:59:26.492605785 (14h20m1.007708349s) [upload=14742]: Got next segment. Inline: false Aug 7 23:59:26.492681402 (14h20m1.007783987s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.492689520 (14h20m1.007792101s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:56.459090933 (14h20m30.974193512s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:56.459165231 (14h20m30.974267796s) [upload=14742]: Done uploading segments... Aug 7 23:59:56.459181138 (14h20m30.974283703s) [upload=14742]: Upload finished. err: context deadline exceeded Aug 7 23:59:56.572495425 (14h20m31.087597994s) [upload=14751]: Starting upload Aug 7 23:59:56.572578962 (14h20m31.087681544s) [upload=14751]: Done starting upload Aug 7 23:59:56.572818459 (14h20m31.087921029s) [upload=14751]: Uploading segments... Aug 7 23:59:56.572953386 (14h20m31.088055953s) [upload=14751]: Got next segment. Inline: false Aug 7 23:59:56.573048904 (14h20m31.088151483s) [upload=14751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:56.573066041 (14h20m31.088168618s) [upload=14751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:26.529103664 (14h21m1.044206252s) [upload=14751]: Next segment err: context deadline exceeded Aug 8 00:00:26.529241226 (14h21m1.044343811s) [upload=14751]: Done uploading segments... Aug 8 00:00:26.529265086 (14h21m1.044367654s) [upload=14751]: Upload finished. err: context deadline exceeded Duration: 472.001968ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [...................X.XX.X.....X..X..........................] duration: 234.354114ms FAIL (2025-08-08 00:21:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 234.378029ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [....................X..XXX..X...X...........................] duration: 237.406575ms FAIL (2025-08-08 00:21:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 237.438024ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X.XXX...X.......................] duration: 1.114536672s FAIL (2025-08-08 00:21:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 474260 (14h23m36.87657683s) [upload=14802]: Got next segment. Inline: false Aug 8 00:03:02.361540620 (14h23m36.876643199s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.361546434 (14h23m36.87664901s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:57.203920880 (14h24m31.719023446s) [seg_pos={0 0}, upload=14802]: Begin upload segment... Aug 8 00:03:57.203988004 (14h24m31.719090583s) [seg_pos={0 0}, upload=14802]: Done begin upload segment. Aug 8 00:03:57.204020830 (14h24m31.719123417s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204041644 (14h24m31.71914423s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204477526 (14h24m31.719580107s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204519100 (14h24m31.719621687s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204895465 (14h24m31.719998052s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204925306 (14h24m31.720027871s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205328523 (14h24m31.720431105s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205769139 (14h24m31.720871721s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207436252 (14h24m31.722538839s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207828410 (14h24m31.722930995s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.208468343 (14h24m31.723570931s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.209378344 (14h24m31.72448093s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.210191405 (14h24m31.72529399s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211289253 (14h24m31.726391837s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211675380 (14h24m31.726777962s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212135467 (14h24m31.727238053s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212506215 (14h24m31.727608796s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212830381 (14h24m31.727932969s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.294294492 (14h24m31.809397074s) [upload=14802]: Next returned nil segment Aug 8 00:03:57.294311485 (14h24m31.809414071s) [upload=14802]: Waiting for error group managing segments... Aug 8 00:03:57.306352633 (14h24m31.821455213s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.307346446 (14h24m31.822449009s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.308700400 (14h24m31.823802978s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.312160131 (14h24m31.827262707s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.320102036 (14h24m31.835204602s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.321164433 (14h24m31.836267014s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.326337358 (14h24m31.841439935s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.352532224 (14h24m31.867634786s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.356261323 (14h24m31.871363886s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.423088525 (14h24m31.938191096s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.471455369 (14h24m31.986557939s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.749384271 (14h24m32.264486838s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.873513358 (14h24m32.388615936s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876598294 (14h24m32.391700872s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876652272 (14h24m32.391754851s) [seg_pos={0 0}, upload=14802]: Segment reached optimal threshold of 14 pieces. Aug 8 00:03:57.877023576 (14h24m32.392126154s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy): protocol: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection Aug 8 00:03:57.877160473 (14h24m32.392263043s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877284905 (14h24m32.392387475s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877449504 (14h24m32.392552073s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ctLLD5skk4t5i3BFYgoTmuZ2aqsX23wDGDMnKriQ7U2KXyPAo): protocol: write tcp 159.223.143.39:47368->104.36.82.3:4794: use of closed network connection; context canceled Aug 8 00:03:57.877469652 (14h24m32.392572221s) [seg_pos={0 0}, upload=14802]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:03:57.877571490 (14h24m32.392674067s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:03:57.877576879 (14h24m32.392679458s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:02.023423762 (14h24m36.538526336s) [upload=14802]: Done uploading segments... Aug 8 00:04:02.023447427 (14h24m36.538550004s) [upload=14802]: Upload finished. err: <nil> Duration: 1.11457435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X..........X............................] duration: 503.948332ms FAIL (2025-08-08 00:21:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.314326ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................X..X....XX...X.........................] duration: 257.161016ms FAIL (2025-08-08 00:21:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 257.192268ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.013017962s FAIL (2025-08-08 00:21:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.013049162s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........................XX..XX.............................] duration: 739.659899ms FAIL (2025-08-08 00:20:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 739.686291ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [.........................XXXXXXX.X..........................] duration: 230.472627ms FAIL (2025-08-08 00:20:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ne starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:22.831162336 (14h14m57.346264913s) [upload=14661]: Starting upload Aug 7 23:54:22.831234431 (14h14m57.346337008s) [upload=14661]: Done starting upload Aug 7 23:54:22.831306794 (14h14m57.346409362s) [upload=14661]: Uploading segments... Aug 7 23:54:22.831343257 (14h14m57.346445842s) [upload=14661]: Got next segment. Inline: true Aug 7 23:54:22.831347808 (14h14m57.346450375s) [upload=14661]: Waiting for error group managing segments... Aug 7 23:54:22.831399492 (14h14m57.346502071s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:22.831404349 (14h14m57.346506915s) [upload=14661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:22.831407696 (14h14m57.346510265s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:22.799123829 (14h15m57.314226409s) [upload=14661]: Done uploading segments... Aug 7 23:55:22.799154191 (14h15m57.314256772s) [upload=14661]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:22.918485444 (14h15m57.433588022s) [upload=14677]: Starting upload Aug 7 23:55:22.918566621 (14h15m57.433672141s) [upload=14677]: Done starting upload Aug 7 23:55:22.918647605 (14h15m57.433750172s) [upload=14677]: Uploading segments... Aug 7 23:55:22.918705315 (14h15m57.4338079s) [upload=14677]: Got next segment. Inline: true Aug 7 23:55:22.918710403 (14h15m57.43381298s) [upload=14677]: Waiting for error group managing segments... Aug 7 23:55:22.918764176 (14h15m57.433866753s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:22.918769239 (14h15m57.433871815s) [upload=14677]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:22.918772352 (14h15m57.433874928s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:22.891103920 (14h16m57.406206502s) [upload=14677]: Done uploading segments... Aug 7 23:56:22.891142892 (14h16m57.406245471s) [upload=14677]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:23.012512156 (14h16m57.527614733s) [upload=14690]: Starting upload Aug 7 23:56:23.012570683 (14h16m57.527673265s) [upload=14690]: Done starting upload Aug 7 23:56:23.012632890 (14h16m57.527735455s) [upload=14690]: Uploading segments... Aug 7 23:56:23.012675514 (14h16m57.527778093s) [upload=14690]: Got next segment. Inline: true Aug 7 23:56:23.012680516 (14h16m57.527783092s) [upload=14690]: Waiting for error group managing segments... Aug 7 23:56:23.012734720 (14h16m57.527837295s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:23.012739356 (14h16m57.527841923s) [upload=14690]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:23.012742484 (14h16m57.527845059s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:22.981126119 (14h17m57.496228686s) [upload=14690]: Done uploading segments... Aug 7 23:57:22.981169317 (14h17m57.496271889s) [upload=14690]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:23.078169780 (14h17m57.593272348s) [upload=14705]: Starting upload Aug 7 23:57:23.078287503 (14h17m57.593390079s) [upload=14705]: Done starting upload Aug 7 23:57:23.078353530 (14h17m57.593456107s) [upload=14705]: Uploading segments... Aug 7 23:57:23.078394403 (14h17m57.593496992s) [upload=14705]: Got next segment. Inline: true Aug 7 23:57:23.078400888 (14h17m57.593503457s) [upload=14705]: Waiting for error group managing segments... Aug 7 23:57:23.078452630 (14h17m57.593555201s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:23.078457270 (14h17m57.593559838s) [upload=14705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:23.078460483 (14h17m57.593563048s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:23.049396934 (14h18m57.564499518s) [upload=14705]: Done uploading segments... Aug 7 23:58:23.049418059 (14h18m57.564520634s) [upload=14705]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.169720285 (14h18m57.684822852s) [upload=14722]: Starting upload Aug 7 23:58:23.169779212 (14h18m57.684881788s) [upload=14722]: Done starting upload Aug 7 23:58:23.169840834 (14h18m57.684943401s) [upload=14722]: Uploading segments... Aug 7 23:58:23.169880437 (14h18m57.684983021s) [upload=14722]: Got next segment. Inline: true Aug 7 23:58:23.169885772 (14h18m57.684988338s) [upload=14722]: Waiting for error group managing segments... Aug 7 23:58:23.169927728 (14h18m57.685030303s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.169933919 (14h18m57.685036498s) [upload=14722]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.169938117 (14h18m57.685040704s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:23.138140642 (14h19m57.653243209s) [upload=14722]: Done uploading segments... Aug 7 23:59:23.138159469 (14h19m57.653262035s) [upload=14722]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:35.502179618 (14h20m10.017282184s) [upload=14745]: Starting upload Aug 7 23:59:35.502246881 (14h20m10.017349449s) [upload=14745]: Done starting upload Aug 7 23:59:35.502308502 (14h20m10.017411071s) [upload=14745]: Uploading segments... Aug 7 23:59:35.502347394 (14h20m10.017449979s) [upload=14745]: Got next segment. Inline: true Aug 7 23:59:35.502351750 (14h20m10.017454327s) [upload=14745]: Waiting for error group managing segments... Aug 7 23:59:35.502390443 (14h20m10.017493009s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:35.502394821 (14h20m10.017497398s) [upload=14745]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:35.502398119 (14h20m10.017500695s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.472447782 (14h21m9.987550364s) [upload=14745]: Done uploading segments... Aug 8 00:00:35.472473966 (14h21m9.987576558s) [upload=14745]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:35.593243192 (14h21m10.10834576s) [upload=14760]: Starting upload Aug 8 00:00:35.593395835 (14h21m10.108498415s) [upload=14760]: Done starting upload Aug 8 00:00:35.593497926 (14h21m10.10860049s) [upload=14760]: Uploading segments... Aug 8 00:00:35.593659481 (14h21m10.108762045s) [upload=14760]: Got next segment. Inline: true Aug 8 00:00:35.593667686 (14h21m10.108770273s) [upload=14760]: Waiting for error group managing segments... Aug 8 00:00:35.593750345 (14h21m10.10885293s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.593773930 (14h21m10.108876508s) [upload=14760]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.593782409 (14h21m10.108884991s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.662793141 (14h21m10.177895709s) [upload=14760]: Done uploading segments... Aug 8 00:00:35.662815087 (14h21m10.177917658s) [upload=14760]: Upload finished. err: <nil> Aug 8 00:01:23.648690375 (14h21m58.163792959s) [upload=14775]: Starting upload Aug 8 00:01:23.648799550 (14h21m58.163902126s) [upload=14775]: Done starting upload Aug 8 00:01:23.648928369 (14h21m58.164030934s) [upload=14775]: Uploading segments... Aug 8 00:01:23.648986999 (14h21m58.164089578s) [upload=14775]: Got next segment. Inline: true Aug 8 00:01:23.648992059 (14h21m58.164094634s) [upload=14775]: Waiting for error group managing segments... Aug 8 00:01:23.649037148 (14h21m58.164139714s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:23.649041722 (14h21m58.164144298s) [upload=14775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:23.649044950 (14h21m58.164147527s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:23.618855090 (14h22m58.133957671s) [upload=14775]: Done uploading segments... Aug 8 00:02:23.618900564 (14h22m58.134003142s) [upload=14775]: Upload finished. err: metaclient: context deadline exceeded Duration: 230.496805ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.448442592s FAIL (2025-08-08 00:20:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== piece... Aug 8 00:06:47.146014029 (14h27m21.661116615s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146042952 (14h27m21.661145531s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146392229 (14h27m21.661494798s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146489907 (14h27m21.661592493s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146751379 (14h27m21.661853965s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146970578 (14h27m21.662073166s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147102314 (14h27m21.662204899s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147440038 (14h27m21.662542625s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147464022 (14h27m21.662566606s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147806500 (14h27m21.662909085s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148072099 (14h27m21.663174685s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148231380 (14h27m21.663333959s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148503333 (14h27m21.663605919s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151364728 (14h27m21.666467312s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151909137 (14h27m21.667011732s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.152324438 (14h27m21.667427024s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.269249030 (14h27m21.7843516s) [upload=14868]: Next returned nil segment Aug 8 00:06:47.269265635 (14h27m21.784368211s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:06:47.280030107 (14h27m21.795132688s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282243133 (14h27m21.797345711s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282935406 (14h27m21.798037984s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.283797129 (14h27m21.798899704s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.288848212 (14h27m21.803950789s) [noise=true , address=45.91.116.5:4301 , piece_id=GQLYE5Z2Q2SHC2WIEZMTPRGYQ7B4EQZZYJLVC4NQMRENIDZATA5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.292885927 (14h27m21.807988496s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.296224440 (14h27m21.811327016s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.299834425 (14h27m21.81493699s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.305883863 (14h27m21.820986439s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.309497417 (14h27m21.824599987s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.323758805 (14h27m21.838861383s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.326822007 (14h27m21.841924593s) [noise=true , address=104.36.82.45:4938 , piece_id=QI275JMAE4DVWGO5CY3IDTC6WKZCNPPCO3LKE5EK74TGJSBLB2VA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.327668872 (14h27m21.842771449s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339100720 (14h27m21.854203297s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339134259 (14h27m21.854236839s) [seg_pos={0 0}, upload=14868]: Segment reached optimal threshold of 14 pieces. Aug 8 00:06:47.339343619 (14h27m21.854446181s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72): protocol: expected piece hash; context canceled; EOF Aug 8 00:06:47.339560288 (14h27m21.854662868s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mTJmjiHwMJqSkuWKrFnAgJUt1gjQ4wPjfgY7qouWcanaNUy46): protocol: read tcp 159.223.143.39:35278->79.127.205.233:4636: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339701256 (14h27m21.854803837s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: read tcp 159.223.143.39:57032->64.49.13.137:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339837792 (14h27m21.854940379s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk): protocol: read tcp 159.223.143.39:48324->64.49.13.27:4968: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:06:47.339968534 (14h27m21.855071118s) [seg_pos={0 0}, upload=14868]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:06:47.340079739 (14h27m21.855182316s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:06:47.340086599 (14h27m21.855189182s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:47.400977058 (14h27m21.916079641s) [upload=14868]: Done uploading segments... Aug 8 00:06:47.401023653 (14h27m21.916126232s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:07:57.606754320 (14h28m32.121856897s) [upload=14881]: Starting upload Aug 8 00:07:57.606822750 (14h28m32.121925318s) [upload=14881]: Done starting upload Aug 8 00:07:57.607097187 (14h28m32.122199756s) [upload=14881]: Uploading segments... Aug 8 00:07:57.607354625 (14h28m32.122457207s) [upload=14881]: Got next segment. Inline: false Aug 8 00:07:57.607435490 (14h28m32.122538073s) [upload=14881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:57.607441733 (14h28m32.122544313s) [upload=14881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:07:57.622249420 (14h28m32.137352005s) [upload=14881]: Next segment err: metaclient: Too Many Requests Aug 8 00:07:57.622307892 (14h28m32.137410469s) [upload=14881]: Done uploading segments... Aug 8 00:07:57.622322946 (14h28m32.137425515s) [upload=14881]: Upload finished. err: metaclient: Too Many Requests Duration: 1.448476188s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................XXX.X...XX.XX............................] duration: 523.340184ms FAIL (2025-08-08 00:20:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130393931 (14h19m30.645496508s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130898433 (14h19m30.646001001s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.131654285 (14h19m30.646756854s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132299167 (14h19m30.647401729s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132815896 (14h19m30.647918459s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.133305230 (14h19m30.648407796s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134064921 (14h19m30.649167503s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134501761 (14h19m30.649604335s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135039838 (14h19m30.650142405s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135450095 (14h19m30.650552665s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135750292 (14h19m30.650852861s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136062113 (14h19m30.651164675s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136413662 (14h19m30.65151623s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136771535 (14h19m30.651874107s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.209688234 (14h19m30.724790803s) [upload=14734]: Next returned nil segment Aug 7 23:58:56.209703014 (14h19m30.72480559s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:58:56.214389537 (14h19m30.729492107s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214445477 (14h19m30.729548054s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214979544 (14h19m30.73008212s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215082462 (14h19m30.730185028s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215826575 (14h19m30.730929141s) [noise=true , address=77.90.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215911546 (14h19m30.731014112s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.216710632 (14h19m30.731813201s) [noise=true , address=77.90.31.18:4807 , piece_id=B3PSWCRJNGDMD6LDDM436N3X75NAA4DEEPLDKL2GTBBJBG3I6ZSQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.217564442 (14h19m30.732667019s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.218100336 (14h19m30.733202913s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.220843127 (14h19m30.735945696s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.221894363 (14h19m30.736996944s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.227829104 (14h19m30.742931668s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326406182 (14h19m30.841508752s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326453057 (14h19m30.841555634s) [seg_pos={0 0}, upload=14734]: Segment reached optimal threshold of 13 pieces. Aug 7 23:58:56.326683605 (14h19m30.841786189s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 23:58:56.326786377 (14h19m30.84188896s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection Aug 7 23:58:56.326888961 (14h19m30.841991543s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: read tcp 159.223.143.39:34618->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:58:56.326913735 (14h19m30.842016315s) [seg_pos={0 0}, upload=14734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:58:56.326998778 (14h19m30.842101358s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:58:56.327004732 (14h19m30.84210731s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:56.375303093 (14h19m30.89040566s) [upload=14734]: Done uploading segments... Aug 7 23:58:56.375321956 (14h19m30.890424523s) [upload=14734]: Upload finished. err: <nil> Aug 7 23:59:26.492160234 (14h20m1.007262811s) [upload=14742]: Starting upload Aug 7 23:59:26.492250238 (14h20m1.007352804s) [upload=14742]: Done starting upload Aug 7 23:59:26.492323252 (14h20m1.00742583s) [upload=14742]: Uploading segments... Aug 7 23:59:26.492605785 (14h20m1.007708349s) [upload=14742]: Got next segment. Inline: false Aug 7 23:59:26.492681402 (14h20m1.007783987s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.492689520 (14h20m1.007792101s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:56.459090933 (14h20m30.974193512s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:56.459165231 (14h20m30.974267796s) [upload=14742]: Done uploading segments... Aug 7 23:59:56.459181138 (14h20m30.974283703s) [upload=14742]: Upload finished. err: context deadline exceeded Aug 7 23:59:56.572495425 (14h20m31.087597994s) [upload=14751]: Starting upload Aug 7 23:59:56.572578962 (14h20m31.087681544s) [upload=14751]: Done starting upload Aug 7 23:59:56.572818459 (14h20m31.087921029s) [upload=14751]: Uploading segments... Aug 7 23:59:56.572953386 (14h20m31.088055953s) [upload=14751]: Got next segment. Inline: false Aug 7 23:59:56.573048904 (14h20m31.088151483s) [upload=14751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:56.573066041 (14h20m31.088168618s) [upload=14751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:26.529103664 (14h21m1.044206252s) [upload=14751]: Next segment err: context deadline exceeded Aug 8 00:00:26.529241226 (14h21m1.044343811s) [upload=14751]: Done uploading segments... Aug 8 00:00:26.529265086 (14h21m1.044367654s) [upload=14751]: Upload finished. err: context deadline exceeded Duration: 523.379536ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 337.650881ms FAIL (2025-08-08 00:20:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== g segments... Aug 8 00:01:46.657245436 (14h22m21.172348015s) [upload=14780]: Upload finished. err: <nil> Aug 8 00:02:30.275266890 (14h23m4.790369467s) [upload=14795]: Starting upload Aug 8 00:02:30.275331708 (14h23m4.790434285s) [upload=14795]: Done starting upload Aug 8 00:02:30.275419113 (14h23m4.790521681s) [upload=14795]: Uploading segments... Aug 8 00:02:30.275464303 (14h23m4.790566902s) [upload=14795]: Got next segment. Inline: true Aug 8 00:02:30.275470409 (14h23m4.790572986s) [upload=14795]: Waiting for error group managing segments... Aug 8 00:02:30.275541468 (14h23m4.790644064s) [upload=14795]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.275548435 (14h23m4.790651025s) [upload=14795]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:30.275553112 (14h23m4.790655694s) [upload=14795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.437129275 (14h23m4.952231852s) [upload=14795]: Done uploading segments... Aug 8 00:02:30.437147837 (14h23m4.952250404s) [upload=14795]: Upload finished. err: <nil> Aug 8 00:03:02.216029168 (14h23m36.731131753s) [upload=14801]: Starting upload Aug 8 00:03:02.216131444 (14h23m36.731234024s) [upload=14801]: Done starting upload Aug 8 00:03:02.216223262 (14h23m36.731325836s) [upload=14801]: Uploading segments... Aug 8 00:03:02.216305099 (14h23m36.731407689s) [upload=14801]: Got next segment. Inline: true Aug 8 00:03:02.216311582 (14h23m36.731414165s) [upload=14801]: Waiting for error group managing segments... Aug 8 00:03:02.216381029 (14h23m36.731483612s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.216386785 (14h23m36.731489374s) [upload=14801]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:02.216390618 (14h23m36.731493197s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:02.283802345 (14h23m36.798904923s) [upload=14801]: Done uploading segments... Aug 8 00:03:02.283823305 (14h23m36.798925889s) [upload=14801]: Upload finished. err: <nil> Aug 8 00:03:30.509760483 (14h24m5.024863047s) [upload=14812]: Starting upload Aug 8 00:03:30.509815323 (14h24m5.024917908s) [upload=14812]: Done starting upload Aug 8 00:03:30.509926692 (14h24m5.02502927s) [upload=14812]: Uploading segments... Aug 8 00:03:30.509996173 (14h24m5.025098754s) [upload=14812]: Got next segment. Inline: true Aug 8 00:03:30.510002796 (14h24m5.025105377s) [upload=14812]: Waiting for error group managing segments... Aug 8 00:03:30.510068440 (14h24m5.025171017s) [upload=14812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:30.510074098 (14h24m5.025176685s) [upload=14812]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:30.510077690 (14h24m5.025180273s) [upload=14812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:30.698239096 (14h24m5.213341663s) [upload=14812]: Done uploading segments... Aug 8 00:03:30.698256740 (14h24m5.213359308s) [upload=14812]: Upload finished. err: <nil> Aug 8 00:04:41.065788390 (14h25m15.580890975s) [upload=14831]: Starting upload Aug 8 00:04:41.065849488 (14h25m15.580952064s) [upload=14831]: Done starting upload Aug 8 00:04:41.065914279 (14h25m15.581016856s) [upload=14831]: Uploading segments... Aug 8 00:04:41.065955528 (14h25m15.581058114s) [upload=14831]: Got next segment. Inline: true Aug 8 00:04:41.065961233 (14h25m15.581063813s) [upload=14831]: Waiting for error group managing segments... Aug 8 00:04:41.066010814 (14h25m15.581113395s) [upload=14831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:04:41.066015217 (14h25m15.581117797s) [upload=14831]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:04:41.066019044 (14h25m15.581121626s) [upload=14831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:41.143560517 (14h25m15.658663093s) [upload=14831]: Done uploading segments... Aug 8 00:04:41.143578273 (14h25m15.658680839s) [upload=14831]: Upload finished. err: <nil> Aug 8 00:04:57.174663208 (14h25m31.689765785s) [upload=14836]: Starting upload Aug 8 00:04:57.174738504 (14h25m31.68984108s) [upload=14836]: Done starting upload Aug 8 00:04:57.174818503 (14h25m31.689921072s) [upload=14836]: Uploading segments... Aug 8 00:04:57.174883379 (14h25m31.689985957s) [upload=14836]: Got next segment. Inline: true Aug 8 00:04:57.174888780 (14h25m31.689991347s) [upload=14836]: Waiting for error group managing segments... Aug 8 00:04:57.174955675 (14h25m31.690058262s) [upload=14836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:04:57.174962095 (14h25m31.69006468s) [upload=14836]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:04:57.174966595 (14h25m31.690069178s) [upload=14836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:57.317214071 (14h25m31.832316638s) [upload=14836]: Done uploading segments... Aug 8 00:04:57.317241163 (14h25m31.832343733s) [upload=14836]: Upload finished. err: <nil> Aug 8 00:06:06.741579891 (14h26m41.256682461s) [upload=14855]: Starting upload Aug 8 00:06:06.741694208 (14h26m41.256796791s) [upload=14855]: Done starting upload Aug 8 00:06:06.741776231 (14h26m41.256878801s) [upload=14855]: Uploading segments... Aug 8 00:06:06.741863808 (14h26m41.256966377s) [upload=14855]: Got next segment. Inline: true Aug 8 00:06:06.741870254 (14h26m41.25697283s) [upload=14855]: Waiting for error group managing segments... Aug 8 00:06:06.741939630 (14h26m41.257042211s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:06:06.741944620 (14h26m41.257047201s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:06:06.741950358 (14h26m41.257052937s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:07.276784579 (14h26m41.791887173s) [upload=14855]: Done uploading segments... Aug 8 00:06:07.276805845 (14h26m41.79190841s) [upload=14855]: Upload finished. err: <nil> Aug 8 00:06:43.194994307 (14h27m17.710096878s) [upload=14865]: Starting upload Aug 8 00:06:43.195063177 (14h27m17.710165753s) [upload=14865]: Done starting upload Aug 8 00:06:43.195131851 (14h27m17.710234419s) [upload=14865]: Uploading segments... Aug 8 00:06:43.195200076 (14h27m17.710302661s) [upload=14865]: Got next segment. Inline: true Aug 8 00:06:43.195205915 (14h27m17.710308491s) [upload=14865]: Waiting for error group managing segments... Aug 8 00:06:43.195271114 (14h27m17.710373692s) [upload=14865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:06:43.195276311 (14h27m17.710378889s) [upload=14865]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:06:43.195279806 (14h27m17.710382383s) [upload=14865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:43.430698626 (14h27m17.945801203s) [upload=14865]: Done uploading segments... Aug 8 00:06:43.430715582 (14h27m17.945818158s) [upload=14865]: Upload finished. err: <nil> Aug 8 00:07:00.838748208 (14h27m35.353850791s) [upload=14870]: Starting upload Aug 8 00:07:00.838831814 (14h27m35.353934397s) [upload=14870]: Done starting upload Aug 8 00:07:00.838944876 (14h27m35.354047439s) [upload=14870]: Uploading segments... Aug 8 00:07:00.839014925 (14h27m35.354117501s) [upload=14870]: Got next segment. Inline: true Aug 8 00:07:00.839045878 (14h27m35.354148457s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:07:00.839120971 (14h27m35.354223549s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:00.839126346 (14h27m35.354228923s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:00.839129942 (14h27m35.354232518s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:00.932912415 (14h27m35.448014989s) [upload=14870]: Done uploading segments... Aug 8 00:07:00.932927726 (14h27m35.448030289s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:08:14.384482900 (14h28m48.899585476s) [upload=14888]: Starting upload Aug 8 00:08:14.384593866 (14h28m48.899696446s) [upload=14888]: Done starting upload Aug 8 00:08:14.384673686 (14h28m48.899776254s) [upload=14888]: Uploading segments... Aug 8 00:08:14.384718832 (14h28m48.899821412s) [upload=14888]: Got next segment. Inline: true Aug 8 00:08:14.384723835 (14h28m48.899826412s) [upload=14888]: Waiting for error group managing segments... Aug 8 00:08:14.384763544 (14h28m48.89986612s) [upload=14888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:14.384768731 (14h28m48.899871307s) [upload=14888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:14.384771994 (14h28m48.89987457s) [upload=14888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:14.710769235 (14h28m49.225871818s) [upload=14888]: Done uploading segments... Aug 8 00:08:14.710803190 (14h28m49.225905767s) [upload=14888]: Upload finished. err: <nil> Aug 8 00:09:18.634609803 (14h29m53.149712376s) [upload=14909]: Starting upload Aug 8 00:09:18.634680614 (14h29m53.149783189s) [upload=14909]: Done starting upload Aug 8 00:09:18.634765976 (14h29m53.149868543s) [upload=14909]: Uploading segments... Aug 8 00:09:18.634815721 (14h29m53.1499183s) [upload=14909]: Got next segment. Inline: true Aug 8 00:09:18.634820719 (14h29m53.149923286s) [upload=14909]: Waiting for error group managing segments... Aug 8 00:09:18.634863856 (14h29m53.149966435s) [upload=14909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:09:18.634868286 (14h29m53.149970863s) [upload=14909]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:09:18.634872291 (14h29m53.14997487s) [upload=14909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:09:18.642683461 (14h29m53.157786047s) [upload=14909]: Done uploading segments... Aug 8 00:09:18.642703785 (14h29m53.157806368s) [upload=14909]: Upload finished. err: metaclient: Too Many Requests Duration: 337.682448ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................X..XXX..X...X..........................] duration: 234.299177ms FAIL (2025-08-08 00:20:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 234.327696ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [....................X.XX.X.....X..X.........................] duration: 384.20412ms FAIL (2025-08-08 00:20:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 384.231109ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.943084384s FAIL (2025-08-08 00:20:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.94312033s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..........X...........................] duration: 520.664823ms FAIL (2025-08-08 00:20:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 520.716019ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................X.XXX...X......................] duration: 961.632401ms FAIL (2025-08-08 00:20:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 474260 (14h23m36.87657683s) [upload=14802]: Got next segment. Inline: false Aug 8 00:03:02.361540620 (14h23m36.876643199s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.361546434 (14h23m36.87664901s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:57.203920880 (14h24m31.719023446s) [seg_pos={0 0}, upload=14802]: Begin upload segment... Aug 8 00:03:57.203988004 (14h24m31.719090583s) [seg_pos={0 0}, upload=14802]: Done begin upload segment. Aug 8 00:03:57.204020830 (14h24m31.719123417s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204041644 (14h24m31.71914423s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204477526 (14h24m31.719580107s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204519100 (14h24m31.719621687s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204895465 (14h24m31.719998052s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204925306 (14h24m31.720027871s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205328523 (14h24m31.720431105s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205769139 (14h24m31.720871721s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207436252 (14h24m31.722538839s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207828410 (14h24m31.722930995s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.208468343 (14h24m31.723570931s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.209378344 (14h24m31.72448093s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.210191405 (14h24m31.72529399s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211289253 (14h24m31.726391837s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211675380 (14h24m31.726777962s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212135467 (14h24m31.727238053s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212506215 (14h24m31.727608796s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212830381 (14h24m31.727932969s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.294294492 (14h24m31.809397074s) [upload=14802]: Next returned nil segment Aug 8 00:03:57.294311485 (14h24m31.809414071s) [upload=14802]: Waiting for error group managing segments... Aug 8 00:03:57.306352633 (14h24m31.821455213s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.307346446 (14h24m31.822449009s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.308700400 (14h24m31.823802978s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.312160131 (14h24m31.827262707s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.320102036 (14h24m31.835204602s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.321164433 (14h24m31.836267014s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.326337358 (14h24m31.841439935s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.352532224 (14h24m31.867634786s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.356261323 (14h24m31.871363886s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.423088525 (14h24m31.938191096s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.471455369 (14h24m31.986557939s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.749384271 (14h24m32.264486838s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.873513358 (14h24m32.388615936s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876598294 (14h24m32.391700872s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876652272 (14h24m32.391754851s) [seg_pos={0 0}, upload=14802]: Segment reached optimal threshold of 14 pieces. Aug 8 00:03:57.877023576 (14h24m32.392126154s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy): protocol: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection Aug 8 00:03:57.877160473 (14h24m32.392263043s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877284905 (14h24m32.392387475s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877449504 (14h24m32.392552073s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ctLLD5skk4t5i3BFYgoTmuZ2aqsX23wDGDMnKriQ7U2KXyPAo): protocol: write tcp 159.223.143.39:47368->104.36.82.3:4794: use of closed network connection; context canceled Aug 8 00:03:57.877469652 (14h24m32.392572221s) [seg_pos={0 0}, upload=14802]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:03:57.877571490 (14h24m32.392674067s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:03:57.877576879 (14h24m32.392679458s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:02.023423762 (14h24m36.538526336s) [upload=14802]: Done uploading segments... Aug 8 00:04:02.023447427 (14h24m36.538550004s) [upload=14802]: Upload finished. err: <nil> Duration: 961.665074ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................XXX.X...XX.XX...........................] duration: 495.693991ms FAIL (2025-08-08 00:20:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130393931 (14h19m30.645496508s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130898433 (14h19m30.646001001s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.131654285 (14h19m30.646756854s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132299167 (14h19m30.647401729s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132815896 (14h19m30.647918459s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.133305230 (14h19m30.648407796s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134064921 (14h19m30.649167503s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134501761 (14h19m30.649604335s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135039838 (14h19m30.650142405s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135450095 (14h19m30.650552665s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135750292 (14h19m30.650852861s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136062113 (14h19m30.651164675s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136413662 (14h19m30.65151623s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136771535 (14h19m30.651874107s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.209688234 (14h19m30.724790803s) [upload=14734]: Next returned nil segment Aug 7 23:58:56.209703014 (14h19m30.72480559s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:58:56.214389537 (14h19m30.729492107s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214445477 (14h19m30.729548054s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214979544 (14h19m30.73008212s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215082462 (14h19m30.730185028s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215826575 (14h19m30.730929141s) [noise=true , address=77.90.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215911546 (14h19m30.731014112s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.216710632 (14h19m30.731813201s) [noise=true , address=77.90.31.18:4807 , piece_id=B3PSWCRJNGDMD6LDDM436N3X75NAA4DEEPLDKL2GTBBJBG3I6ZSQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.217564442 (14h19m30.732667019s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.218100336 (14h19m30.733202913s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.220843127 (14h19m30.735945696s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.221894363 (14h19m30.736996944s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.227829104 (14h19m30.742931668s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326406182 (14h19m30.841508752s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326453057 (14h19m30.841555634s) [seg_pos={0 0}, upload=14734]: Segment reached optimal threshold of 13 pieces. Aug 7 23:58:56.326683605 (14h19m30.841786189s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 23:58:56.326786377 (14h19m30.84188896s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection Aug 7 23:58:56.326888961 (14h19m30.841991543s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: read tcp 159.223.143.39:34618->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:58:56.326913735 (14h19m30.842016315s) [seg_pos={0 0}, upload=14734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:58:56.326998778 (14h19m30.842101358s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:58:56.327004732 (14h19m30.84210731s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:56.375303093 (14h19m30.89040566s) [upload=14734]: Done uploading segments... Aug 7 23:58:56.375321956 (14h19m30.890424523s) [upload=14734]: Upload finished. err: <nil> Aug 7 23:59:26.492160234 (14h20m1.007262811s) [upload=14742]: Starting upload Aug 7 23:59:26.492250238 (14h20m1.007352804s) [upload=14742]: Done starting upload Aug 7 23:59:26.492323252 (14h20m1.00742583s) [upload=14742]: Uploading segments... Aug 7 23:59:26.492605785 (14h20m1.007708349s) [upload=14742]: Got next segment. Inline: false Aug 7 23:59:26.492681402 (14h20m1.007783987s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.492689520 (14h20m1.007792101s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:56.459090933 (14h20m30.974193512s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:56.459165231 (14h20m30.974267796s) [upload=14742]: Done uploading segments... Aug 7 23:59:56.459181138 (14h20m30.974283703s) [upload=14742]: Upload finished. err: context deadline exceeded Aug 7 23:59:56.572495425 (14h20m31.087597994s) [upload=14751]: Starting upload Aug 7 23:59:56.572578962 (14h20m31.087681544s) [upload=14751]: Done starting upload Aug 7 23:59:56.572818459 (14h20m31.087921029s) [upload=14751]: Uploading segments... Aug 7 23:59:56.572953386 (14h20m31.088055953s) [upload=14751]: Got next segment. Inline: false Aug 7 23:59:56.573048904 (14h20m31.088151483s) [upload=14751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:56.573066041 (14h20m31.088168618s) [upload=14751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:26.529103664 (14h21m1.044206252s) [upload=14751]: Next segment err: context deadline exceeded Aug 8 00:00:26.529241226 (14h21m1.044343811s) [upload=14751]: Done uploading segments... Aug 8 00:00:26.529265086 (14h21m1.044367654s) [upload=14751]: Upload finished. err: context deadline exceeded Duration: 495.731112ms core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 79.364857ms target: https://us1.storj.io/login FAIL (2025-08-08 00:20:04 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 79.386242ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 651.574933ms FAIL (2025-08-08 00:20:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== piece... Aug 8 00:06:47.146014029 (14h27m21.661116615s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146042952 (14h27m21.661145531s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146392229 (14h27m21.661494798s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146489907 (14h27m21.661592493s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146751379 (14h27m21.661853965s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146970578 (14h27m21.662073166s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147102314 (14h27m21.662204899s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147440038 (14h27m21.662542625s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147464022 (14h27m21.662566606s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147806500 (14h27m21.662909085s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148072099 (14h27m21.663174685s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148231380 (14h27m21.663333959s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148503333 (14h27m21.663605919s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151364728 (14h27m21.666467312s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151909137 (14h27m21.667011732s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.152324438 (14h27m21.667427024s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.269249030 (14h27m21.7843516s) [upload=14868]: Next returned nil segment Aug 8 00:06:47.269265635 (14h27m21.784368211s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:06:47.280030107 (14h27m21.795132688s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282243133 (14h27m21.797345711s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282935406 (14h27m21.798037984s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.283797129 (14h27m21.798899704s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.288848212 (14h27m21.803950789s) [noise=true , address=45.91.116.5:4301 , piece_id=GQLYE5Z2Q2SHC2WIEZMTPRGYQ7B4EQZZYJLVC4NQMRENIDZATA5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.292885927 (14h27m21.807988496s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.296224440 (14h27m21.811327016s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.299834425 (14h27m21.81493699s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.305883863 (14h27m21.820986439s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.309497417 (14h27m21.824599987s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.323758805 (14h27m21.838861383s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.326822007 (14h27m21.841924593s) [noise=true , address=104.36.82.45:4938 , piece_id=QI275JMAE4DVWGO5CY3IDTC6WKZCNPPCO3LKE5EK74TGJSBLB2VA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.327668872 (14h27m21.842771449s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339100720 (14h27m21.854203297s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339134259 (14h27m21.854236839s) [seg_pos={0 0}, upload=14868]: Segment reached optimal threshold of 14 pieces. Aug 8 00:06:47.339343619 (14h27m21.854446181s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72): protocol: expected piece hash; context canceled; EOF Aug 8 00:06:47.339560288 (14h27m21.854662868s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mTJmjiHwMJqSkuWKrFnAgJUt1gjQ4wPjfgY7qouWcanaNUy46): protocol: read tcp 159.223.143.39:35278->79.127.205.233:4636: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339701256 (14h27m21.854803837s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: read tcp 159.223.143.39:57032->64.49.13.137:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339837792 (14h27m21.854940379s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk): protocol: read tcp 159.223.143.39:48324->64.49.13.27:4968: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:06:47.339968534 (14h27m21.855071118s) [seg_pos={0 0}, upload=14868]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:06:47.340079739 (14h27m21.855182316s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:06:47.340086599 (14h27m21.855189182s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:47.400977058 (14h27m21.916079641s) [upload=14868]: Done uploading segments... Aug 8 00:06:47.401023653 (14h27m21.916126232s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:07:57.606754320 (14h28m32.121856897s) [upload=14881]: Starting upload Aug 8 00:07:57.606822750 (14h28m32.121925318s) [upload=14881]: Done starting upload Aug 8 00:07:57.607097187 (14h28m32.122199756s) [upload=14881]: Uploading segments... Aug 8 00:07:57.607354625 (14h28m32.122457207s) [upload=14881]: Got next segment. Inline: false Aug 8 00:07:57.607435490 (14h28m32.122538073s) [upload=14881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:57.607441733 (14h28m32.122544313s) [upload=14881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:07:57.622249420 (14h28m32.137352005s) [upload=14881]: Next segment err: metaclient: Too Many Requests Aug 8 00:07:57.622307892 (14h28m32.137410469s) [upload=14881]: Done uploading segments... Aug 8 00:07:57.622322946 (14h28m32.137425515s) [upload=14881]: Upload finished. err: metaclient: Too Many Requests Duration: 651.634296ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................X..X....XX...X........................] duration: 308.846204ms FAIL (2025-08-08 00:20:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 308.87446ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................X.XX.X.....X..X........................] duration: 212.938933ms FAIL (2025-08-08 00:19:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 212.994047ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................X..XXX..X...X.........................] duration: 223.962304ms FAIL (2025-08-08 00:19:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 223.989524ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.960784551s FAIL (2025-08-08 00:19:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.960813425s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........................XX..XX............................] duration: 742.574607ms FAIL (2025-08-08 00:19:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 742.597137ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [..........................XXXXXXX.X.........................] duration: 253.295093ms FAIL (2025-08-08 00:19:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ne starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:22.831162336 (14h14m57.346264913s) [upload=14661]: Starting upload Aug 7 23:54:22.831234431 (14h14m57.346337008s) [upload=14661]: Done starting upload Aug 7 23:54:22.831306794 (14h14m57.346409362s) [upload=14661]: Uploading segments... Aug 7 23:54:22.831343257 (14h14m57.346445842s) [upload=14661]: Got next segment. Inline: true Aug 7 23:54:22.831347808 (14h14m57.346450375s) [upload=14661]: Waiting for error group managing segments... Aug 7 23:54:22.831399492 (14h14m57.346502071s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:22.831404349 (14h14m57.346506915s) [upload=14661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:22.831407696 (14h14m57.346510265s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:22.799123829 (14h15m57.314226409s) [upload=14661]: Done uploading segments... Aug 7 23:55:22.799154191 (14h15m57.314256772s) [upload=14661]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:22.918485444 (14h15m57.433588022s) [upload=14677]: Starting upload Aug 7 23:55:22.918566621 (14h15m57.433672141s) [upload=14677]: Done starting upload Aug 7 23:55:22.918647605 (14h15m57.433750172s) [upload=14677]: Uploading segments... Aug 7 23:55:22.918705315 (14h15m57.4338079s) [upload=14677]: Got next segment. Inline: true Aug 7 23:55:22.918710403 (14h15m57.43381298s) [upload=14677]: Waiting for error group managing segments... Aug 7 23:55:22.918764176 (14h15m57.433866753s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:22.918769239 (14h15m57.433871815s) [upload=14677]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:22.918772352 (14h15m57.433874928s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:22.891103920 (14h16m57.406206502s) [upload=14677]: Done uploading segments... Aug 7 23:56:22.891142892 (14h16m57.406245471s) [upload=14677]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:23.012512156 (14h16m57.527614733s) [upload=14690]: Starting upload Aug 7 23:56:23.012570683 (14h16m57.527673265s) [upload=14690]: Done starting upload Aug 7 23:56:23.012632890 (14h16m57.527735455s) [upload=14690]: Uploading segments... Aug 7 23:56:23.012675514 (14h16m57.527778093s) [upload=14690]: Got next segment. Inline: true Aug 7 23:56:23.012680516 (14h16m57.527783092s) [upload=14690]: Waiting for error group managing segments... Aug 7 23:56:23.012734720 (14h16m57.527837295s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:23.012739356 (14h16m57.527841923s) [upload=14690]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:23.012742484 (14h16m57.527845059s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:22.981126119 (14h17m57.496228686s) [upload=14690]: Done uploading segments... Aug 7 23:57:22.981169317 (14h17m57.496271889s) [upload=14690]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:23.078169780 (14h17m57.593272348s) [upload=14705]: Starting upload Aug 7 23:57:23.078287503 (14h17m57.593390079s) [upload=14705]: Done starting upload Aug 7 23:57:23.078353530 (14h17m57.593456107s) [upload=14705]: Uploading segments... Aug 7 23:57:23.078394403 (14h17m57.593496992s) [upload=14705]: Got next segment. Inline: true Aug 7 23:57:23.078400888 (14h17m57.593503457s) [upload=14705]: Waiting for error group managing segments... Aug 7 23:57:23.078452630 (14h17m57.593555201s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:23.078457270 (14h17m57.593559838s) [upload=14705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:23.078460483 (14h17m57.593563048s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:23.049396934 (14h18m57.564499518s) [upload=14705]: Done uploading segments... Aug 7 23:58:23.049418059 (14h18m57.564520634s) [upload=14705]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.169720285 (14h18m57.684822852s) [upload=14722]: Starting upload Aug 7 23:58:23.169779212 (14h18m57.684881788s) [upload=14722]: Done starting upload Aug 7 23:58:23.169840834 (14h18m57.684943401s) [upload=14722]: Uploading segments... Aug 7 23:58:23.169880437 (14h18m57.684983021s) [upload=14722]: Got next segment. Inline: true Aug 7 23:58:23.169885772 (14h18m57.684988338s) [upload=14722]: Waiting for error group managing segments... Aug 7 23:58:23.169927728 (14h18m57.685030303s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.169933919 (14h18m57.685036498s) [upload=14722]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.169938117 (14h18m57.685040704s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:23.138140642 (14h19m57.653243209s) [upload=14722]: Done uploading segments... Aug 7 23:59:23.138159469 (14h19m57.653262035s) [upload=14722]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:35.502179618 (14h20m10.017282184s) [upload=14745]: Starting upload Aug 7 23:59:35.502246881 (14h20m10.017349449s) [upload=14745]: Done starting upload Aug 7 23:59:35.502308502 (14h20m10.017411071s) [upload=14745]: Uploading segments... Aug 7 23:59:35.502347394 (14h20m10.017449979s) [upload=14745]: Got next segment. Inline: true Aug 7 23:59:35.502351750 (14h20m10.017454327s) [upload=14745]: Waiting for error group managing segments... Aug 7 23:59:35.502390443 (14h20m10.017493009s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:35.502394821 (14h20m10.017497398s) [upload=14745]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:35.502398119 (14h20m10.017500695s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.472447782 (14h21m9.987550364s) [upload=14745]: Done uploading segments... Aug 8 00:00:35.472473966 (14h21m9.987576558s) [upload=14745]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:35.593243192 (14h21m10.10834576s) [upload=14760]: Starting upload Aug 8 00:00:35.593395835 (14h21m10.108498415s) [upload=14760]: Done starting upload Aug 8 00:00:35.593497926 (14h21m10.10860049s) [upload=14760]: Uploading segments... Aug 8 00:00:35.593659481 (14h21m10.108762045s) [upload=14760]: Got next segment. Inline: true Aug 8 00:00:35.593667686 (14h21m10.108770273s) [upload=14760]: Waiting for error group managing segments... Aug 8 00:00:35.593750345 (14h21m10.10885293s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.593773930 (14h21m10.108876508s) [upload=14760]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.593782409 (14h21m10.108884991s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.662793141 (14h21m10.177895709s) [upload=14760]: Done uploading segments... Aug 8 00:00:35.662815087 (14h21m10.177917658s) [upload=14760]: Upload finished. err: <nil> Aug 8 00:01:23.648690375 (14h21m58.163792959s) [upload=14775]: Starting upload Aug 8 00:01:23.648799550 (14h21m58.163902126s) [upload=14775]: Done starting upload Aug 8 00:01:23.648928369 (14h21m58.164030934s) [upload=14775]: Uploading segments... Aug 8 00:01:23.648986999 (14h21m58.164089578s) [upload=14775]: Got next segment. Inline: true Aug 8 00:01:23.648992059 (14h21m58.164094634s) [upload=14775]: Waiting for error group managing segments... Aug 8 00:01:23.649037148 (14h21m58.164139714s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:23.649041722 (14h21m58.164144298s) [upload=14775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:23.649044950 (14h21m58.164147527s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:23.618855090 (14h22m58.133957671s) [upload=14775]: Done uploading segments... Aug 8 00:02:23.618900564 (14h22m58.134003142s) [upload=14775]: Upload finished. err: metaclient: context deadline exceeded Duration: 253.318103ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................X.XX.X.....X..X.......................] duration: 233.795908ms FAIL (2025-08-08 00:19:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 233.824447ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.813782046s FAIL (2025-08-08 00:19:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.813813181s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [...........................XXXXXXX.X........................] duration: 239.625504ms FAIL (2025-08-08 00:19:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ne starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:22.831162336 (14h14m57.346264913s) [upload=14661]: Starting upload Aug 7 23:54:22.831234431 (14h14m57.346337008s) [upload=14661]: Done starting upload Aug 7 23:54:22.831306794 (14h14m57.346409362s) [upload=14661]: Uploading segments... Aug 7 23:54:22.831343257 (14h14m57.346445842s) [upload=14661]: Got next segment. Inline: true Aug 7 23:54:22.831347808 (14h14m57.346450375s) [upload=14661]: Waiting for error group managing segments... Aug 7 23:54:22.831399492 (14h14m57.346502071s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:22.831404349 (14h14m57.346506915s) [upload=14661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:22.831407696 (14h14m57.346510265s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:22.799123829 (14h15m57.314226409s) [upload=14661]: Done uploading segments... Aug 7 23:55:22.799154191 (14h15m57.314256772s) [upload=14661]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:22.918485444 (14h15m57.433588022s) [upload=14677]: Starting upload Aug 7 23:55:22.918566621 (14h15m57.433672141s) [upload=14677]: Done starting upload Aug 7 23:55:22.918647605 (14h15m57.433750172s) [upload=14677]: Uploading segments... Aug 7 23:55:22.918705315 (14h15m57.4338079s) [upload=14677]: Got next segment. Inline: true Aug 7 23:55:22.918710403 (14h15m57.43381298s) [upload=14677]: Waiting for error group managing segments... Aug 7 23:55:22.918764176 (14h15m57.433866753s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:22.918769239 (14h15m57.433871815s) [upload=14677]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:22.918772352 (14h15m57.433874928s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:22.891103920 (14h16m57.406206502s) [upload=14677]: Done uploading segments... Aug 7 23:56:22.891142892 (14h16m57.406245471s) [upload=14677]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:23.012512156 (14h16m57.527614733s) [upload=14690]: Starting upload Aug 7 23:56:23.012570683 (14h16m57.527673265s) [upload=14690]: Done starting upload Aug 7 23:56:23.012632890 (14h16m57.527735455s) [upload=14690]: Uploading segments... Aug 7 23:56:23.012675514 (14h16m57.527778093s) [upload=14690]: Got next segment. Inline: true Aug 7 23:56:23.012680516 (14h16m57.527783092s) [upload=14690]: Waiting for error group managing segments... Aug 7 23:56:23.012734720 (14h16m57.527837295s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:23.012739356 (14h16m57.527841923s) [upload=14690]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:23.012742484 (14h16m57.527845059s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:22.981126119 (14h17m57.496228686s) [upload=14690]: Done uploading segments... Aug 7 23:57:22.981169317 (14h17m57.496271889s) [upload=14690]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:23.078169780 (14h17m57.593272348s) [upload=14705]: Starting upload Aug 7 23:57:23.078287503 (14h17m57.593390079s) [upload=14705]: Done starting upload Aug 7 23:57:23.078353530 (14h17m57.593456107s) [upload=14705]: Uploading segments... Aug 7 23:57:23.078394403 (14h17m57.593496992s) [upload=14705]: Got next segment. Inline: true Aug 7 23:57:23.078400888 (14h17m57.593503457s) [upload=14705]: Waiting for error group managing segments... Aug 7 23:57:23.078452630 (14h17m57.593555201s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:23.078457270 (14h17m57.593559838s) [upload=14705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:23.078460483 (14h17m57.593563048s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:23.049396934 (14h18m57.564499518s) [upload=14705]: Done uploading segments... Aug 7 23:58:23.049418059 (14h18m57.564520634s) [upload=14705]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.169720285 (14h18m57.684822852s) [upload=14722]: Starting upload Aug 7 23:58:23.169779212 (14h18m57.684881788s) [upload=14722]: Done starting upload Aug 7 23:58:23.169840834 (14h18m57.684943401s) [upload=14722]: Uploading segments... Aug 7 23:58:23.169880437 (14h18m57.684983021s) [upload=14722]: Got next segment. Inline: true Aug 7 23:58:23.169885772 (14h18m57.684988338s) [upload=14722]: Waiting for error group managing segments... Aug 7 23:58:23.169927728 (14h18m57.685030303s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.169933919 (14h18m57.685036498s) [upload=14722]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.169938117 (14h18m57.685040704s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:23.138140642 (14h19m57.653243209s) [upload=14722]: Done uploading segments... Aug 7 23:59:23.138159469 (14h19m57.653262035s) [upload=14722]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:35.502179618 (14h20m10.017282184s) [upload=14745]: Starting upload Aug 7 23:59:35.502246881 (14h20m10.017349449s) [upload=14745]: Done starting upload Aug 7 23:59:35.502308502 (14h20m10.017411071s) [upload=14745]: Uploading segments... Aug 7 23:59:35.502347394 (14h20m10.017449979s) [upload=14745]: Got next segment. Inline: true Aug 7 23:59:35.502351750 (14h20m10.017454327s) [upload=14745]: Waiting for error group managing segments... Aug 7 23:59:35.502390443 (14h20m10.017493009s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:35.502394821 (14h20m10.017497398s) [upload=14745]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:35.502398119 (14h20m10.017500695s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.472447782 (14h21m9.987550364s) [upload=14745]: Done uploading segments... Aug 8 00:00:35.472473966 (14h21m9.987576558s) [upload=14745]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:35.593243192 (14h21m10.10834576s) [upload=14760]: Starting upload Aug 8 00:00:35.593395835 (14h21m10.108498415s) [upload=14760]: Done starting upload Aug 8 00:00:35.593497926 (14h21m10.10860049s) [upload=14760]: Uploading segments... Aug 8 00:00:35.593659481 (14h21m10.108762045s) [upload=14760]: Got next segment. Inline: true Aug 8 00:00:35.593667686 (14h21m10.108770273s) [upload=14760]: Waiting for error group managing segments... Aug 8 00:00:35.593750345 (14h21m10.10885293s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.593773930 (14h21m10.108876508s) [upload=14760]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.593782409 (14h21m10.108884991s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.662793141 (14h21m10.177895709s) [upload=14760]: Done uploading segments... Aug 8 00:00:35.662815087 (14h21m10.177917658s) [upload=14760]: Upload finished. err: <nil> Aug 8 00:01:23.648690375 (14h21m58.163792959s) [upload=14775]: Starting upload Aug 8 00:01:23.648799550 (14h21m58.163902126s) [upload=14775]: Done starting upload Aug 8 00:01:23.648928369 (14h21m58.164030934s) [upload=14775]: Uploading segments... Aug 8 00:01:23.648986999 (14h21m58.164089578s) [upload=14775]: Got next segment. Inline: true Aug 8 00:01:23.648992059 (14h21m58.164094634s) [upload=14775]: Waiting for error group managing segments... Aug 8 00:01:23.649037148 (14h21m58.164139714s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:23.649041722 (14h21m58.164144298s) [upload=14775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:23.649044950 (14h21m58.164147527s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:23.618855090 (14h22m58.133957671s) [upload=14775]: Done uploading segments... Aug 8 00:02:23.618900564 (14h22m58.134003142s) [upload=14775]: Upload finished. err: metaclient: context deadline exceeded Duration: 239.714152ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 736.734249ms FAIL (2025-08-08 00:19:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== g segments... Aug 8 00:01:46.657245436 (14h22m21.172348015s) [upload=14780]: Upload finished. err: <nil> Aug 8 00:02:30.275266890 (14h23m4.790369467s) [upload=14795]: Starting upload Aug 8 00:02:30.275331708 (14h23m4.790434285s) [upload=14795]: Done starting upload Aug 8 00:02:30.275419113 (14h23m4.790521681s) [upload=14795]: Uploading segments... Aug 8 00:02:30.275464303 (14h23m4.790566902s) [upload=14795]: Got next segment. Inline: true Aug 8 00:02:30.275470409 (14h23m4.790572986s) [upload=14795]: Waiting for error group managing segments... Aug 8 00:02:30.275541468 (14h23m4.790644064s) [upload=14795]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.275548435 (14h23m4.790651025s) [upload=14795]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:30.275553112 (14h23m4.790655694s) [upload=14795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:30.437129275 (14h23m4.952231852s) [upload=14795]: Done uploading segments... Aug 8 00:02:30.437147837 (14h23m4.952250404s) [upload=14795]: Upload finished. err: <nil> Aug 8 00:03:02.216029168 (14h23m36.731131753s) [upload=14801]: Starting upload Aug 8 00:03:02.216131444 (14h23m36.731234024s) [upload=14801]: Done starting upload Aug 8 00:03:02.216223262 (14h23m36.731325836s) [upload=14801]: Uploading segments... Aug 8 00:03:02.216305099 (14h23m36.731407689s) [upload=14801]: Got next segment. Inline: true Aug 8 00:03:02.216311582 (14h23m36.731414165s) [upload=14801]: Waiting for error group managing segments... Aug 8 00:03:02.216381029 (14h23m36.731483612s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.216386785 (14h23m36.731489374s) [upload=14801]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:02.216390618 (14h23m36.731493197s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:02.283802345 (14h23m36.798904923s) [upload=14801]: Done uploading segments... Aug 8 00:03:02.283823305 (14h23m36.798925889s) [upload=14801]: Upload finished. err: <nil> Aug 8 00:03:30.509760483 (14h24m5.024863047s) [upload=14812]: Starting upload Aug 8 00:03:30.509815323 (14h24m5.024917908s) [upload=14812]: Done starting upload Aug 8 00:03:30.509926692 (14h24m5.02502927s) [upload=14812]: Uploading segments... Aug 8 00:03:30.509996173 (14h24m5.025098754s) [upload=14812]: Got next segment. Inline: true Aug 8 00:03:30.510002796 (14h24m5.025105377s) [upload=14812]: Waiting for error group managing segments... Aug 8 00:03:30.510068440 (14h24m5.025171017s) [upload=14812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:30.510074098 (14h24m5.025176685s) [upload=14812]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:30.510077690 (14h24m5.025180273s) [upload=14812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:30.698239096 (14h24m5.213341663s) [upload=14812]: Done uploading segments... Aug 8 00:03:30.698256740 (14h24m5.213359308s) [upload=14812]: Upload finished. err: <nil> Aug 8 00:04:41.065788390 (14h25m15.580890975s) [upload=14831]: Starting upload Aug 8 00:04:41.065849488 (14h25m15.580952064s) [upload=14831]: Done starting upload Aug 8 00:04:41.065914279 (14h25m15.581016856s) [upload=14831]: Uploading segments... Aug 8 00:04:41.065955528 (14h25m15.581058114s) [upload=14831]: Got next segment. Inline: true Aug 8 00:04:41.065961233 (14h25m15.581063813s) [upload=14831]: Waiting for error group managing segments... Aug 8 00:04:41.066010814 (14h25m15.581113395s) [upload=14831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:04:41.066015217 (14h25m15.581117797s) [upload=14831]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:04:41.066019044 (14h25m15.581121626s) [upload=14831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:41.143560517 (14h25m15.658663093s) [upload=14831]: Done uploading segments... Aug 8 00:04:41.143578273 (14h25m15.658680839s) [upload=14831]: Upload finished. err: <nil> Aug 8 00:04:57.174663208 (14h25m31.689765785s) [upload=14836]: Starting upload Aug 8 00:04:57.174738504 (14h25m31.68984108s) [upload=14836]: Done starting upload Aug 8 00:04:57.174818503 (14h25m31.689921072s) [upload=14836]: Uploading segments... Aug 8 00:04:57.174883379 (14h25m31.689985957s) [upload=14836]: Got next segment. Inline: true Aug 8 00:04:57.174888780 (14h25m31.689991347s) [upload=14836]: Waiting for error group managing segments... Aug 8 00:04:57.174955675 (14h25m31.690058262s) [upload=14836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:04:57.174962095 (14h25m31.69006468s) [upload=14836]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:04:57.174966595 (14h25m31.690069178s) [upload=14836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:57.317214071 (14h25m31.832316638s) [upload=14836]: Done uploading segments... Aug 8 00:04:57.317241163 (14h25m31.832343733s) [upload=14836]: Upload finished. err: <nil> Aug 8 00:06:06.741579891 (14h26m41.256682461s) [upload=14855]: Starting upload Aug 8 00:06:06.741694208 (14h26m41.256796791s) [upload=14855]: Done starting upload Aug 8 00:06:06.741776231 (14h26m41.256878801s) [upload=14855]: Uploading segments... Aug 8 00:06:06.741863808 (14h26m41.256966377s) [upload=14855]: Got next segment. Inline: true Aug 8 00:06:06.741870254 (14h26m41.25697283s) [upload=14855]: Waiting for error group managing segments... Aug 8 00:06:06.741939630 (14h26m41.257042211s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:06:06.741944620 (14h26m41.257047201s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:06:06.741950358 (14h26m41.257052937s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:07.276784579 (14h26m41.791887173s) [upload=14855]: Done uploading segments... Aug 8 00:06:07.276805845 (14h26m41.79190841s) [upload=14855]: Upload finished. err: <nil> Aug 8 00:06:43.194994307 (14h27m17.710096878s) [upload=14865]: Starting upload Aug 8 00:06:43.195063177 (14h27m17.710165753s) [upload=14865]: Done starting upload Aug 8 00:06:43.195131851 (14h27m17.710234419s) [upload=14865]: Uploading segments... Aug 8 00:06:43.195200076 (14h27m17.710302661s) [upload=14865]: Got next segment. Inline: true Aug 8 00:06:43.195205915 (14h27m17.710308491s) [upload=14865]: Waiting for error group managing segments... Aug 8 00:06:43.195271114 (14h27m17.710373692s) [upload=14865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:06:43.195276311 (14h27m17.710378889s) [upload=14865]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:06:43.195279806 (14h27m17.710382383s) [upload=14865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:43.430698626 (14h27m17.945801203s) [upload=14865]: Done uploading segments... Aug 8 00:06:43.430715582 (14h27m17.945818158s) [upload=14865]: Upload finished. err: <nil> Aug 8 00:07:00.838748208 (14h27m35.353850791s) [upload=14870]: Starting upload Aug 8 00:07:00.838831814 (14h27m35.353934397s) [upload=14870]: Done starting upload Aug 8 00:07:00.838944876 (14h27m35.354047439s) [upload=14870]: Uploading segments... Aug 8 00:07:00.839014925 (14h27m35.354117501s) [upload=14870]: Got next segment. Inline: true Aug 8 00:07:00.839045878 (14h27m35.354148457s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:07:00.839120971 (14h27m35.354223549s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:00.839126346 (14h27m35.354228923s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:07:00.839129942 (14h27m35.354232518s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:07:00.932912415 (14h27m35.448014989s) [upload=14870]: Done uploading segments... Aug 8 00:07:00.932927726 (14h27m35.448030289s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:08:14.384482900 (14h28m48.899585476s) [upload=14888]: Starting upload Aug 8 00:08:14.384593866 (14h28m48.899696446s) [upload=14888]: Done starting upload Aug 8 00:08:14.384673686 (14h28m48.899776254s) [upload=14888]: Uploading segments... Aug 8 00:08:14.384718832 (14h28m48.899821412s) [upload=14888]: Got next segment. Inline: true Aug 8 00:08:14.384723835 (14h28m48.899826412s) [upload=14888]: Waiting for error group managing segments... Aug 8 00:08:14.384763544 (14h28m48.89986612s) [upload=14888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:08:14.384768731 (14h28m48.899871307s) [upload=14888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:08:14.384771994 (14h28m48.89987457s) [upload=14888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:08:14.710769235 (14h28m49.225871818s) [upload=14888]: Done uploading segments... Aug 8 00:08:14.710803190 (14h28m49.225905767s) [upload=14888]: Upload finished. err: <nil> Aug 8 00:09:18.634609803 (14h29m53.149712376s) [upload=14909]: Starting upload Aug 8 00:09:18.634680614 (14h29m53.149783189s) [upload=14909]: Done starting upload Aug 8 00:09:18.634765976 (14h29m53.149868543s) [upload=14909]: Uploading segments... Aug 8 00:09:18.634815721 (14h29m53.1499183s) [upload=14909]: Got next segment. Inline: true Aug 8 00:09:18.634820719 (14h29m53.149923286s) [upload=14909]: Waiting for error group managing segments... Aug 8 00:09:18.634863856 (14h29m53.149966435s) [upload=14909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:09:18.634868286 (14h29m53.149970863s) [upload=14909]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:09:18.634872291 (14h29m53.14997487s) [upload=14909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:09:18.642683461 (14h29m53.157786047s) [upload=14909]: Done uploading segments... Aug 8 00:09:18.642703785 (14h29m53.157806368s) [upload=14909]: Upload finished. err: metaclient: Too Many Requests Duration: 736.759953ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................X..X....XX...X.......................] duration: 266.363729ms FAIL (2025-08-08 00:19:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 266.399194ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.......................X.XX.X.....X..X......................] duration: 280.547996ms FAIL (2025-08-08 00:19:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 280.574949ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................XXX.X...XX.XX..........................] duration: 363.285712ms FAIL (2025-08-08 00:19:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130393931 (14h19m30.645496508s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130898433 (14h19m30.646001001s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.131654285 (14h19m30.646756854s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132299167 (14h19m30.647401729s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132815896 (14h19m30.647918459s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.133305230 (14h19m30.648407796s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134064921 (14h19m30.649167503s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134501761 (14h19m30.649604335s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135039838 (14h19m30.650142405s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135450095 (14h19m30.650552665s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135750292 (14h19m30.650852861s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136062113 (14h19m30.651164675s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136413662 (14h19m30.65151623s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136771535 (14h19m30.651874107s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.209688234 (14h19m30.724790803s) [upload=14734]: Next returned nil segment Aug 7 23:58:56.209703014 (14h19m30.72480559s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:58:56.214389537 (14h19m30.729492107s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214445477 (14h19m30.729548054s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214979544 (14h19m30.73008212s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215082462 (14h19m30.730185028s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215826575 (14h19m30.730929141s) [noise=true , address=77.90.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215911546 (14h19m30.731014112s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.216710632 (14h19m30.731813201s) [noise=true , address=77.90.31.18:4807 , piece_id=B3PSWCRJNGDMD6LDDM436N3X75NAA4DEEPLDKL2GTBBJBG3I6ZSQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.217564442 (14h19m30.732667019s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.218100336 (14h19m30.733202913s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.220843127 (14h19m30.735945696s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.221894363 (14h19m30.736996944s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.227829104 (14h19m30.742931668s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326406182 (14h19m30.841508752s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326453057 (14h19m30.841555634s) [seg_pos={0 0}, upload=14734]: Segment reached optimal threshold of 13 pieces. Aug 7 23:58:56.326683605 (14h19m30.841786189s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 23:58:56.326786377 (14h19m30.84188896s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection Aug 7 23:58:56.326888961 (14h19m30.841991543s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: read tcp 159.223.143.39:34618->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:58:56.326913735 (14h19m30.842016315s) [seg_pos={0 0}, upload=14734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:58:56.326998778 (14h19m30.842101358s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:58:56.327004732 (14h19m30.84210731s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:56.375303093 (14h19m30.89040566s) [upload=14734]: Done uploading segments... Aug 7 23:58:56.375321956 (14h19m30.890424523s) [upload=14734]: Upload finished. err: <nil> Aug 7 23:59:26.492160234 (14h20m1.007262811s) [upload=14742]: Starting upload Aug 7 23:59:26.492250238 (14h20m1.007352804s) [upload=14742]: Done starting upload Aug 7 23:59:26.492323252 (14h20m1.00742583s) [upload=14742]: Uploading segments... Aug 7 23:59:26.492605785 (14h20m1.007708349s) [upload=14742]: Got next segment. Inline: false Aug 7 23:59:26.492681402 (14h20m1.007783987s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.492689520 (14h20m1.007792101s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:56.459090933 (14h20m30.974193512s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:56.459165231 (14h20m30.974267796s) [upload=14742]: Done uploading segments... Aug 7 23:59:56.459181138 (14h20m30.974283703s) [upload=14742]: Upload finished. err: context deadline exceeded Aug 7 23:59:56.572495425 (14h20m31.087597994s) [upload=14751]: Starting upload Aug 7 23:59:56.572578962 (14h20m31.087681544s) [upload=14751]: Done starting upload Aug 7 23:59:56.572818459 (14h20m31.087921029s) [upload=14751]: Uploading segments... Aug 7 23:59:56.572953386 (14h20m31.088055953s) [upload=14751]: Got next segment. Inline: false Aug 7 23:59:56.573048904 (14h20m31.088151483s) [upload=14751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:56.573066041 (14h20m31.088168618s) [upload=14751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:26.529103664 (14h21m1.044206252s) [upload=14751]: Next segment err: context deadline exceeded Aug 8 00:00:26.529241226 (14h21m1.044343811s) [upload=14751]: Done uploading segments... Aug 8 00:00:26.529265086 (14h21m1.044367654s) [upload=14751]: Upload finished. err: context deadline exceeded Duration: 363.323538ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X.XXX...X.....................] duration: 620.824537ms FAIL (2025-08-08 00:19:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 474260 (14h23m36.87657683s) [upload=14802]: Got next segment. Inline: false Aug 8 00:03:02.361540620 (14h23m36.876643199s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.361546434 (14h23m36.87664901s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:57.203920880 (14h24m31.719023446s) [seg_pos={0 0}, upload=14802]: Begin upload segment... Aug 8 00:03:57.203988004 (14h24m31.719090583s) [seg_pos={0 0}, upload=14802]: Done begin upload segment. Aug 8 00:03:57.204020830 (14h24m31.719123417s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204041644 (14h24m31.71914423s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204477526 (14h24m31.719580107s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204519100 (14h24m31.719621687s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204895465 (14h24m31.719998052s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204925306 (14h24m31.720027871s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205328523 (14h24m31.720431105s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205769139 (14h24m31.720871721s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207436252 (14h24m31.722538839s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207828410 (14h24m31.722930995s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.208468343 (14h24m31.723570931s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.209378344 (14h24m31.72448093s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.210191405 (14h24m31.72529399s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211289253 (14h24m31.726391837s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211675380 (14h24m31.726777962s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212135467 (14h24m31.727238053s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212506215 (14h24m31.727608796s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212830381 (14h24m31.727932969s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.294294492 (14h24m31.809397074s) [upload=14802]: Next returned nil segment Aug 8 00:03:57.294311485 (14h24m31.809414071s) [upload=14802]: Waiting for error group managing segments... Aug 8 00:03:57.306352633 (14h24m31.821455213s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.307346446 (14h24m31.822449009s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.308700400 (14h24m31.823802978s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.312160131 (14h24m31.827262707s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.320102036 (14h24m31.835204602s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.321164433 (14h24m31.836267014s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.326337358 (14h24m31.841439935s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.352532224 (14h24m31.867634786s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.356261323 (14h24m31.871363886s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.423088525 (14h24m31.938191096s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.471455369 (14h24m31.986557939s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.749384271 (14h24m32.264486838s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.873513358 (14h24m32.388615936s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876598294 (14h24m32.391700872s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876652272 (14h24m32.391754851s) [seg_pos={0 0}, upload=14802]: Segment reached optimal threshold of 14 pieces. Aug 8 00:03:57.877023576 (14h24m32.392126154s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy): protocol: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection Aug 8 00:03:57.877160473 (14h24m32.392263043s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877284905 (14h24m32.392387475s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877449504 (14h24m32.392552073s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ctLLD5skk4t5i3BFYgoTmuZ2aqsX23wDGDMnKriQ7U2KXyPAo): protocol: write tcp 159.223.143.39:47368->104.36.82.3:4794: use of closed network connection; context canceled Aug 8 00:03:57.877469652 (14h24m32.392572221s) [seg_pos={0 0}, upload=14802]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:03:57.877571490 (14h24m32.392674067s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:03:57.877576879 (14h24m32.392679458s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:02.023423762 (14h24m36.538526336s) [upload=14802]: Done uploading segments... Aug 8 00:04:02.023447427 (14h24m36.538550004s) [upload=14802]: Upload finished. err: <nil> Duration: 620.856121ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................XX..XX...........................] duration: 687.179672ms FAIL (2025-08-08 00:19:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 687.203797ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)