Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X..........X.....] duration: 547.031701ms FAIL (2025-08-08 05:20:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== TNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.396619078 (19h36m18.911721655s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397045508 (19h36m18.91214809s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397160558 (19h36m18.912263143s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397451153 (19h36m18.912553734s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397679365 (19h36m18.912781931s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397898222 (19h36m18.9130008s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398125648 (19h36m18.913228218s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398278438 (19h36m18.913381022s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398652749 (19h36m18.913755329s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399000971 (19h36m18.914103551s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399455504 (19h36m18.914558137s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399717456 (19h36m18.914820039s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399736184 (19h36m18.914838767s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399780848 (19h36m18.914883425s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400090065 (19h36m18.915192652s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400430542 (19h36m18.915533108s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.401080440 (19h36m18.916183005s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.492815163 (19h36m19.007917733s) [upload=20259]: Next returned nil segment Aug 8 05:15:44.492830313 (19h36m19.007932879s) [upload=20259]: Waiting for error group managing segments... Aug 8 05:15:44.503634225 (19h36m19.018736809s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504718521 (19h36m19.019821102s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504983984 (19h36m19.020086552s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505253948 (19h36m19.020356524s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505859429 (19h36m19.020962003s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506188048 (19h36m19.021290625s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506966629 (19h36m19.022069192s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507255656 (19h36m19.022358218s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507914747 (19h36m19.02301733s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.508934794 (19h36m19.024037376s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509462525 (19h36m19.024565093s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509503916 (19h36m19.02460648s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509998107 (19h36m19.025100676s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510133131 (19h36m19.025235694s) [noise=true , address=104.36.82.2:4764 , piece_id=4QZF2DZ36ONDTNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510142680 (19h36m19.025245251s) [seg_pos={0 0}, upload=20259]: Segment reached optimal threshold of 14 pieces. Aug 8 05:15:44.510939805 (19h36m19.026042372s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WKWWwa3svATpoNY1z7LNvYmA8gA6ZFpJsk17G4c37tDJVJ2wj): protocol: expected piece hash; context canceled; EOF Aug 8 05:15:44.511262091 (19h36m19.026364669s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.511421518 (19h36m19.026524102s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: read tcp 159.223.143.39:32802->64.49.13.169:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511435693 (19h36m19.026538256s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TEiTVsNxZYEYXCnfZ8h4UiEqCBRBPih3YP5cnEZEMTYoiPEqC): protocol: read tcp 159.223.143.39:49428->64.49.13.154:4934: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511449276 (19h36m19.026551856s) [seg_pos={0 0}, upload=20259]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:15:44.511530782 (19h36m19.026633369s) [upload=20259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:15:44.511536651 (19h36m19.026639235s) [upload=20259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:15:44.590856427 (19h36m19.105959005s) [upload=20259]: Done uploading segments... Aug 8 05:15:44.590921713 (19h36m19.106024292s) [upload=20259]: Upload finished. err: <nil> Aug 8 05:16:32.714498510 (19h37m7.229601094s) [upload=20272]: Starting upload Aug 8 05:16:32.714552893 (19h37m7.229655473s) [upload=20272]: Done starting upload Aug 8 05:16:32.714903852 (19h37m7.230006423s) [upload=20272]: Uploading segments... Aug 8 05:16:32.714955229 (19h37m7.230057809s) [upload=20272]: Got next segment. Inline: false Aug 8 05:16:32.714989454 (19h37m7.230092036s) [upload=20272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:16:32.714994640 (19h37m7.230097224s) [upload=20272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:16:32.723976411 (19h37m7.239078998s) [upload=20272]: Next segment err: metaclient: Too Many Requests Aug 8 05:16:32.723996518 (19h37m7.239099103s) [upload=20272]: Done uploading segments... Aug 8 05:16:32.724012336 (19h37m7.239114909s) [upload=20272]: Upload finished. err: metaclient: Too Many Requests Duration: 547.05351ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 623.556481ms FAIL (2025-08-08 05:20:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 623.583605ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 215.307752ms FAIL (2025-08-08 05:20:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 215.360783ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 27.910028ms target: https://us1.storj.io/login FAIL (2025-08-08 05:19:56 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 27.931435ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 219.392646ms FAIL (2025-08-08 05:19:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 219.417035ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X..........X....] duration: 436.261106ms FAIL (2025-08-08 05:19:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== TNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.396619078 (19h36m18.911721655s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397045508 (19h36m18.91214809s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397160558 (19h36m18.912263143s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397451153 (19h36m18.912553734s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397679365 (19h36m18.912781931s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397898222 (19h36m18.9130008s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398125648 (19h36m18.913228218s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398278438 (19h36m18.913381022s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398652749 (19h36m18.913755329s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399000971 (19h36m18.914103551s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399455504 (19h36m18.914558137s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399717456 (19h36m18.914820039s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399736184 (19h36m18.914838767s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399780848 (19h36m18.914883425s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400090065 (19h36m18.915192652s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400430542 (19h36m18.915533108s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.401080440 (19h36m18.916183005s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.492815163 (19h36m19.007917733s) [upload=20259]: Next returned nil segment Aug 8 05:15:44.492830313 (19h36m19.007932879s) [upload=20259]: Waiting for error group managing segments... Aug 8 05:15:44.503634225 (19h36m19.018736809s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504718521 (19h36m19.019821102s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504983984 (19h36m19.020086552s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505253948 (19h36m19.020356524s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505859429 (19h36m19.020962003s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506188048 (19h36m19.021290625s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506966629 (19h36m19.022069192s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507255656 (19h36m19.022358218s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507914747 (19h36m19.02301733s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.508934794 (19h36m19.024037376s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509462525 (19h36m19.024565093s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509503916 (19h36m19.02460648s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509998107 (19h36m19.025100676s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510133131 (19h36m19.025235694s) [noise=true , address=104.36.82.2:4764 , piece_id=4QZF2DZ36ONDTNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510142680 (19h36m19.025245251s) [seg_pos={0 0}, upload=20259]: Segment reached optimal threshold of 14 pieces. Aug 8 05:15:44.510939805 (19h36m19.026042372s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WKWWwa3svATpoNY1z7LNvYmA8gA6ZFpJsk17G4c37tDJVJ2wj): protocol: expected piece hash; context canceled; EOF Aug 8 05:15:44.511262091 (19h36m19.026364669s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.511421518 (19h36m19.026524102s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: read tcp 159.223.143.39:32802->64.49.13.169:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511435693 (19h36m19.026538256s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TEiTVsNxZYEYXCnfZ8h4UiEqCBRBPih3YP5cnEZEMTYoiPEqC): protocol: read tcp 159.223.143.39:49428->64.49.13.154:4934: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511449276 (19h36m19.026551856s) [seg_pos={0 0}, upload=20259]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:15:44.511530782 (19h36m19.026633369s) [upload=20259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:15:44.511536651 (19h36m19.026639235s) [upload=20259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:15:44.590856427 (19h36m19.105959005s) [upload=20259]: Done uploading segments... Aug 8 05:15:44.590921713 (19h36m19.106024292s) [upload=20259]: Upload finished. err: <nil> Aug 8 05:16:32.714498510 (19h37m7.229601094s) [upload=20272]: Starting upload Aug 8 05:16:32.714552893 (19h37m7.229655473s) [upload=20272]: Done starting upload Aug 8 05:16:32.714903852 (19h37m7.230006423s) [upload=20272]: Uploading segments... Aug 8 05:16:32.714955229 (19h37m7.230057809s) [upload=20272]: Got next segment. Inline: false Aug 8 05:16:32.714989454 (19h37m7.230092036s) [upload=20272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:16:32.714994640 (19h37m7.230097224s) [upload=20272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:16:32.723976411 (19h37m7.239078998s) [upload=20272]: Next segment err: metaclient: Too Many Requests Aug 8 05:16:32.723996518 (19h37m7.239099103s) [upload=20272]: Done uploading segments... Aug 8 05:16:32.724012336 (19h37m7.239114909s) [upload=20272]: Upload finished. err: metaclient: Too Many Requests Duration: 436.295095ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 642.738783ms FAIL (2025-08-08 05:19:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 643.448866ms edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 30.023423ms target: https://us1.storj.io/login FAIL (2025-08-08 05:18:51 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 30.046633ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 416.293032ms FAIL (2025-08-08 05:18:43 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 416.324526ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 490.891899ms FAIL (2025-08-08 05:18:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 490.957275ms edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 29.034355ms target: https://us1.storj.io/login FAIL (2025-08-08 05:18:36 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 29.056644ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X..........X...] duration: 383.727821ms FAIL (2025-08-08 05:18:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== TNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.396619078 (19h36m18.911721655s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397045508 (19h36m18.91214809s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397160558 (19h36m18.912263143s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397451153 (19h36m18.912553734s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397679365 (19h36m18.912781931s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397898222 (19h36m18.9130008s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398125648 (19h36m18.913228218s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398278438 (19h36m18.913381022s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398652749 (19h36m18.913755329s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399000971 (19h36m18.914103551s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399455504 (19h36m18.914558137s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399717456 (19h36m18.914820039s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399736184 (19h36m18.914838767s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399780848 (19h36m18.914883425s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400090065 (19h36m18.915192652s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400430542 (19h36m18.915533108s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.401080440 (19h36m18.916183005s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.492815163 (19h36m19.007917733s) [upload=20259]: Next returned nil segment Aug 8 05:15:44.492830313 (19h36m19.007932879s) [upload=20259]: Waiting for error group managing segments... Aug 8 05:15:44.503634225 (19h36m19.018736809s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504718521 (19h36m19.019821102s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504983984 (19h36m19.020086552s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505253948 (19h36m19.020356524s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505859429 (19h36m19.020962003s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506188048 (19h36m19.021290625s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506966629 (19h36m19.022069192s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507255656 (19h36m19.022358218s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507914747 (19h36m19.02301733s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.508934794 (19h36m19.024037376s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509462525 (19h36m19.024565093s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509503916 (19h36m19.02460648s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509998107 (19h36m19.025100676s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510133131 (19h36m19.025235694s) [noise=true , address=104.36.82.2:4764 , piece_id=4QZF2DZ36ONDTNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510142680 (19h36m19.025245251s) [seg_pos={0 0}, upload=20259]: Segment reached optimal threshold of 14 pieces. Aug 8 05:15:44.510939805 (19h36m19.026042372s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WKWWwa3svATpoNY1z7LNvYmA8gA6ZFpJsk17G4c37tDJVJ2wj): protocol: expected piece hash; context canceled; EOF Aug 8 05:15:44.511262091 (19h36m19.026364669s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.511421518 (19h36m19.026524102s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: read tcp 159.223.143.39:32802->64.49.13.169:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511435693 (19h36m19.026538256s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TEiTVsNxZYEYXCnfZ8h4UiEqCBRBPih3YP5cnEZEMTYoiPEqC): protocol: read tcp 159.223.143.39:49428->64.49.13.154:4934: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511449276 (19h36m19.026551856s) [seg_pos={0 0}, upload=20259]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:15:44.511530782 (19h36m19.026633369s) [upload=20259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:15:44.511536651 (19h36m19.026639235s) [upload=20259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:15:44.590856427 (19h36m19.105959005s) [upload=20259]: Done uploading segments... Aug 8 05:15:44.590921713 (19h36m19.106024292s) [upload=20259]: Upload finished. err: <nil> Aug 8 05:16:32.714498510 (19h37m7.229601094s) [upload=20272]: Starting upload Aug 8 05:16:32.714552893 (19h37m7.229655473s) [upload=20272]: Done starting upload Aug 8 05:16:32.714903852 (19h37m7.230006423s) [upload=20272]: Uploading segments... Aug 8 05:16:32.714955229 (19h37m7.230057809s) [upload=20272]: Got next segment. Inline: false Aug 8 05:16:32.714989454 (19h37m7.230092036s) [upload=20272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:16:32.714994640 (19h37m7.230097224s) [upload=20272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:16:32.723976411 (19h37m7.239078998s) [upload=20272]: Next segment err: metaclient: Too Many Requests Aug 8 05:16:32.723996518 (19h37m7.239099103s) [upload=20272]: Done uploading segments... Aug 8 05:16:32.724012336 (19h37m7.239114909s) [upload=20272]: Upload finished. err: metaclient: Too Many Requests Duration: 383.78411ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X..........X..] duration: 412.518723ms FAIL (2025-08-08 05:17:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== TNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.396619078 (19h36m18.911721655s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397045508 (19h36m18.91214809s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397160558 (19h36m18.912263143s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397451153 (19h36m18.912553734s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397679365 (19h36m18.912781931s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397898222 (19h36m18.9130008s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398125648 (19h36m18.913228218s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398278438 (19h36m18.913381022s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398652749 (19h36m18.913755329s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399000971 (19h36m18.914103551s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399455504 (19h36m18.914558137s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399717456 (19h36m18.914820039s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399736184 (19h36m18.914838767s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399780848 (19h36m18.914883425s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400090065 (19h36m18.915192652s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400430542 (19h36m18.915533108s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.401080440 (19h36m18.916183005s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.492815163 (19h36m19.007917733s) [upload=20259]: Next returned nil segment Aug 8 05:15:44.492830313 (19h36m19.007932879s) [upload=20259]: Waiting for error group managing segments... Aug 8 05:15:44.503634225 (19h36m19.018736809s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504718521 (19h36m19.019821102s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504983984 (19h36m19.020086552s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505253948 (19h36m19.020356524s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505859429 (19h36m19.020962003s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506188048 (19h36m19.021290625s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506966629 (19h36m19.022069192s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507255656 (19h36m19.022358218s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507914747 (19h36m19.02301733s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.508934794 (19h36m19.024037376s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509462525 (19h36m19.024565093s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509503916 (19h36m19.02460648s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509998107 (19h36m19.025100676s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510133131 (19h36m19.025235694s) [noise=true , address=104.36.82.2:4764 , piece_id=4QZF2DZ36ONDTNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510142680 (19h36m19.025245251s) [seg_pos={0 0}, upload=20259]: Segment reached optimal threshold of 14 pieces. Aug 8 05:15:44.510939805 (19h36m19.026042372s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WKWWwa3svATpoNY1z7LNvYmA8gA6ZFpJsk17G4c37tDJVJ2wj): protocol: expected piece hash; context canceled; EOF Aug 8 05:15:44.511262091 (19h36m19.026364669s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.511421518 (19h36m19.026524102s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: read tcp 159.223.143.39:32802->64.49.13.169:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511435693 (19h36m19.026538256s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TEiTVsNxZYEYXCnfZ8h4UiEqCBRBPih3YP5cnEZEMTYoiPEqC): protocol: read tcp 159.223.143.39:49428->64.49.13.154:4934: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511449276 (19h36m19.026551856s) [seg_pos={0 0}, upload=20259]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:15:44.511530782 (19h36m19.026633369s) [upload=20259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:15:44.511536651 (19h36m19.026639235s) [upload=20259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:15:44.590856427 (19h36m19.105959005s) [upload=20259]: Done uploading segments... Aug 8 05:15:44.590921713 (19h36m19.106024292s) [upload=20259]: Upload finished. err: <nil> Aug 8 05:16:32.714498510 (19h37m7.229601094s) [upload=20272]: Starting upload Aug 8 05:16:32.714552893 (19h37m7.229655473s) [upload=20272]: Done starting upload Aug 8 05:16:32.714903852 (19h37m7.230006423s) [upload=20272]: Uploading segments... Aug 8 05:16:32.714955229 (19h37m7.230057809s) [upload=20272]: Got next segment. Inline: false Aug 8 05:16:32.714989454 (19h37m7.230092036s) [upload=20272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:16:32.714994640 (19h37m7.230097224s) [upload=20272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:16:32.723976411 (19h37m7.239078998s) [upload=20272]: Next segment err: metaclient: Too Many Requests Aug 8 05:16:32.723996518 (19h37m7.239099103s) [upload=20272]: Done uploading segments... Aug 8 05:16:32.724012336 (19h37m7.239114909s) [upload=20272]: Upload finished. err: metaclient: Too Many Requests Duration: 412.556335ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 275.035253ms FAIL (2025-08-08 05:17:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 275.059369ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.062904093s FAIL (2025-08-08 05:17:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.062927313s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 462.179827ms FAIL (2025-08-08 05:17:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 462.202826ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 9.08125ms target: https://us1.storj.io/login FAIL (2025-08-08 05:17:31 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 9.107208ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 427.981593ms FAIL (2025-08-08 05:16:58 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 428.004626ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X..........X.] duration: 424.424965ms FAIL (2025-08-08 05:16:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== TNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.396619078 (19h36m18.911721655s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397045508 (19h36m18.91214809s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397160558 (19h36m18.912263143s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397451153 (19h36m18.912553734s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397679365 (19h36m18.912781931s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397898222 (19h36m18.9130008s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398125648 (19h36m18.913228218s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398278438 (19h36m18.913381022s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398652749 (19h36m18.913755329s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399000971 (19h36m18.914103551s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399455504 (19h36m18.914558137s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399717456 (19h36m18.914820039s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399736184 (19h36m18.914838767s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399780848 (19h36m18.914883425s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400090065 (19h36m18.915192652s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400430542 (19h36m18.915533108s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.401080440 (19h36m18.916183005s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.492815163 (19h36m19.007917733s) [upload=20259]: Next returned nil segment Aug 8 05:15:44.492830313 (19h36m19.007932879s) [upload=20259]: Waiting for error group managing segments... Aug 8 05:15:44.503634225 (19h36m19.018736809s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504718521 (19h36m19.019821102s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504983984 (19h36m19.020086552s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505253948 (19h36m19.020356524s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505859429 (19h36m19.020962003s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506188048 (19h36m19.021290625s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506966629 (19h36m19.022069192s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507255656 (19h36m19.022358218s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507914747 (19h36m19.02301733s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.508934794 (19h36m19.024037376s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509462525 (19h36m19.024565093s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509503916 (19h36m19.02460648s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509998107 (19h36m19.025100676s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510133131 (19h36m19.025235694s) [noise=true , address=104.36.82.2:4764 , piece_id=4QZF2DZ36ONDTNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510142680 (19h36m19.025245251s) [seg_pos={0 0}, upload=20259]: Segment reached optimal threshold of 14 pieces. Aug 8 05:15:44.510939805 (19h36m19.026042372s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WKWWwa3svATpoNY1z7LNvYmA8gA6ZFpJsk17G4c37tDJVJ2wj): protocol: expected piece hash; context canceled; EOF Aug 8 05:15:44.511262091 (19h36m19.026364669s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.511421518 (19h36m19.026524102s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: read tcp 159.223.143.39:32802->64.49.13.169:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511435693 (19h36m19.026538256s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TEiTVsNxZYEYXCnfZ8h4UiEqCBRBPih3YP5cnEZEMTYoiPEqC): protocol: read tcp 159.223.143.39:49428->64.49.13.154:4934: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511449276 (19h36m19.026551856s) [seg_pos={0 0}, upload=20259]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:15:44.511530782 (19h36m19.026633369s) [upload=20259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:15:44.511536651 (19h36m19.026639235s) [upload=20259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:15:44.590856427 (19h36m19.105959005s) [upload=20259]: Done uploading segments... Aug 8 05:15:44.590921713 (19h36m19.106024292s) [upload=20259]: Upload finished. err: <nil> Aug 8 05:16:32.714498510 (19h37m7.229601094s) [upload=20272]: Starting upload Aug 8 05:16:32.714552893 (19h37m7.229655473s) [upload=20272]: Done starting upload Aug 8 05:16:32.714903852 (19h37m7.230006423s) [upload=20272]: Uploading segments... Aug 8 05:16:32.714955229 (19h37m7.230057809s) [upload=20272]: Got next segment. Inline: false Aug 8 05:16:32.714989454 (19h37m7.230092036s) [upload=20272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:16:32.714994640 (19h37m7.230097224s) [upload=20272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:16:32.723976411 (19h37m7.239078998s) [upload=20272]: Next segment err: metaclient: Too Many Requests Aug 8 05:16:32.723996518 (19h37m7.239099103s) [upload=20272]: Done uploading segments... Aug 8 05:16:32.724012336 (19h37m7.239114909s) [upload=20272]: Upload finished. err: metaclient: Too Many Requests Duration: 424.532224ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X..........X] duration: 42.068939ms FAIL (2025-08-08 05:16:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== TNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.396619078 (19h36m18.911721655s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397045508 (19h36m18.91214809s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397160558 (19h36m18.912263143s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397451153 (19h36m18.912553734s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397679365 (19h36m18.912781931s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.397898222 (19h36m18.9130008s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398125648 (19h36m18.913228218s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398278438 (19h36m18.913381022s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.398652749 (19h36m18.913755329s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399000971 (19h36m18.914103551s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399455504 (19h36m18.914558137s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399717456 (19h36m18.914820039s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399736184 (19h36m18.914838767s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.399780848 (19h36m18.914883425s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400090065 (19h36m18.915192652s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.400430542 (19h36m18.915533108s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.401080440 (19h36m18.916183005s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Uploading piece... Aug 8 05:15:44.492815163 (19h36m19.007917733s) [upload=20259]: Next returned nil segment Aug 8 05:15:44.492830313 (19h36m19.007932879s) [upload=20259]: Waiting for error group managing segments... Aug 8 05:15:44.503634225 (19h36m19.018736809s) [noise=true , address=104.36.82.58:4298 , piece_id=KANX4YVAKK7RYWKQKUXRGMEBPYREPUZECM7OTPP6GBEG6I57XZ7Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504718521 (19h36m19.019821102s) [noise=true , address=104.36.82.23:4460 , piece_id=3UHZQH52DDCDDWHK6C65ZJHKTKL2YUZUW3V2M7URKGTYOOJIALLA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.504983984 (19h36m19.020086552s) [noise=true , address=104.36.82.60:4349 , piece_id=L6BGCU7M6BX74OPTBCSQBJ3PMOBRN3W562T7EVR3DD6UECY67IRQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505253948 (19h36m19.020356524s) [noise=true , address=104.36.82.45:4928 , piece_id=KHGBLGWNAAG5722YT6XP6U2FA2GK7WJE7MT2RNMGFPYXSRFEIM3Q, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.505859429 (19h36m19.020962003s) [noise=true , address=104.36.82.34:4573 , piece_id=KEGNFSTAEF5YBDXO32577DD7IFWZNSZG5SDSPJFTDOKPWMU7NVFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506188048 (19h36m19.021290625s) [noise=true , address=104.36.82.46:4493 , piece_id=HNGMHYYJCUEF5WGYT356AEKZDY6IZUJX5WN5JVJ36NSC5IEC6AGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.506966629 (19h36m19.022069192s) [noise=true , address=104.36.82.43:4840 , piece_id=R73RV4DJOGZSDDPV3QSXMTKTJ6MQR72XBNXYAEJG5L7I2E43U4HQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507255656 (19h36m19.022358218s) [noise=true , address=104.36.82.37:4655 , piece_id=FULT7QYOBIRP5SGZ54HWYDHET74J5UJAYYUOC5AYE5IBWRS6ZBFQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.507914747 (19h36m19.02301733s) [noise=true , address=104.36.82.53:4123 , piece_id=MDKE7CTOUIFOC4ENQQ6RXJW6FIEMKRHW2GVIRFUBT7G22WUSXCNA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.508934794 (19h36m19.024037376s) [noise=true , address=104.36.82.8:4984 , piece_id=NGMFUDZN4M2ONJQDH5EML2LRX7WBHSZLMLEHU6335YVFJ6TK7CCQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509462525 (19h36m19.024565093s) [noise=true , address=104.36.82.12:4180 , piece_id=ILG74TZ5BFRDRVYKVPPM2VOVOZ4IXFX4RILK3Y5QFKT7W6J6PXIQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509503916 (19h36m19.02460648s) [noise=true , address=104.36.82.62:4421 , piece_id=GWLO2QUJEPUYB5SKGXYY32HNVTQTFM7LLJNOO7WIJOSYKYMXDJMQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.509998107 (19h36m19.025100676s) [noise=true , address=104.36.82.47:4529 , piece_id=BRL5VAKSFPTIWSBEC3N5ZZFL5YBYA4WTMPG4PN7SPQCLF2KOHDJQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510133131 (19h36m19.025235694s) [noise=true , address=104.36.82.2:4764 , piece_id=4QZF2DZ36ONDTNVS3YHCRQE5AP3GTOWQZJD2F2IXHRHIJZCKZAGA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.510142680 (19h36m19.025245251s) [seg_pos={0 0}, upload=20259]: Segment reached optimal threshold of 14 pieces. Aug 8 05:15:44.510939805 (19h36m19.026042372s) [noise=true , address=104.36.82.61:4385 , piece_id=XWK5KD27JAWA7MRDIG5OXUUXBLVY72AVDBR2JWOKUNRUON5BC6LA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WKWWwa3svATpoNY1z7LNvYmA8gA6ZFpJsk17G4c37tDJVJ2wj): protocol: expected piece hash; context canceled; EOF Aug 8 05:15:44.511262091 (19h36m19.026364669s) [noise=true , address=104.36.82.17:4360 , piece_id=TVVJIC7A5UTX3TLYJZ4XEEYLUFWK5JRETR4J2MW6SKYISP7J44SA, seg_pos={0 0}, upload=20259]: Done uploading piece. err: <nil> Aug 8 05:15:44.511421518 (19h36m19.026524102s) [noise=true , address=64.49.13.169:4707 , piece_id=WTQM4UP3B655VDT55DX26KE7WMJP7QYVBWWUHH7PALIX75HBN2ZQ, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: read tcp 159.223.143.39:32802->64.49.13.169:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511435693 (19h36m19.026538256s) [noise=true , address=64.49.13.154:4934 , piece_id=B5JFP2WFQW65PPA6C7SOO6STYYYQKEANV4Z5UDFDGWWIUAWMUN3A, seg_pos={0 0}, upload=20259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TEiTVsNxZYEYXCnfZ8h4UiEqCBRBPih3YP5cnEZEMTYoiPEqC): protocol: read tcp 159.223.143.39:49428->64.49.13.154:4934: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:15:44.511449276 (19h36m19.026551856s) [seg_pos={0 0}, upload=20259]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:15:44.511530782 (19h36m19.026633369s) [upload=20259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:15:44.511536651 (19h36m19.026639235s) [upload=20259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:15:44.590856427 (19h36m19.105959005s) [upload=20259]: Done uploading segments... Aug 8 05:15:44.590921713 (19h36m19.106024292s) [upload=20259]: Upload finished. err: <nil> Aug 8 05:16:32.714498510 (19h37m7.229601094s) [upload=20272]: Starting upload Aug 8 05:16:32.714552893 (19h37m7.229655473s) [upload=20272]: Done starting upload Aug 8 05:16:32.714903852 (19h37m7.230006423s) [upload=20272]: Uploading segments... Aug 8 05:16:32.714955229 (19h37m7.230057809s) [upload=20272]: Got next segment. Inline: false Aug 8 05:16:32.714989454 (19h37m7.230092036s) [upload=20272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:16:32.714994640 (19h37m7.230097224s) [upload=20272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:16:32.723976411 (19h37m7.239078998s) [upload=20272]: Next segment err: metaclient: Too Many Requests Aug 8 05:16:32.723996518 (19h37m7.239099103s) [upload=20272]: Done uploading segments... Aug 8 05:16:32.724012336 (19h37m7.239114909s) [upload=20272]: Upload finished. err: metaclient: Too Many Requests Duration: 42.091867ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 558.005104ms FAIL (2025-08-08 05:16:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 558.04405ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 171.646248ms FAIL (2025-08-08 05:16:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 171.67696ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 485.649042ms FAIL (2025-08-08 05:16:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 485.671937ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 650.413544ms FAIL (2025-08-08 05:15:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 823884 (19h28m25.204926455s) [seg_pos={0 0}, upload=20117]: Done begin upload segment. Aug 8 05:07:50.689845352 (19h28m25.204947931s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.689876284 (19h28m25.20497887s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690299079 (19h28m25.205401961s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690414644 (19h28m25.205517222s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690692455 (19h28m25.205795037s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690753877 (19h28m25.205856454s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691078732 (19h28m25.206181312s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691274182 (19h28m25.206376764s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691315491 (19h28m25.206418073s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691540668 (19h28m25.20664325s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691764660 (19h28m25.206867235s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691991240 (19h28m25.207093819s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692208677 (19h28m25.207311255s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692483009 (19h28m25.207585591s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692631937 (19h28m25.20773452s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692956125 (19h28m25.208058727s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693275480 (19h28m25.208378064s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693517734 (19h28m25.208620316s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.796953768 (19h28m25.312056349s) [upload=20117]: Next returned nil segment Aug 8 05:07:50.796969939 (19h28m25.312072524s) [upload=20117]: Waiting for error group managing segments... Aug 8 05:07:50.807280798 (19h28m25.322383368s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.807499572 (19h28m25.32260214s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808409851 (19h28m25.323512419s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808537380 (19h28m25.32363996s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809142026 (19h28m25.324244594s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809727637 (19h28m25.324830208s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810036385 (19h28m25.325138952s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810524084 (19h28m25.325626646s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811239458 (19h28m25.326342021s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811747718 (19h28m25.326850288s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812233806 (19h28m25.327336381s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812577779 (19h28m25.327680346s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813745244 (19h28m25.328847814s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813953245 (19h28m25.329055807s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813966177 (19h28m25.329068739s) [seg_pos={0 0}, upload=20117]: Segment reached optimal threshold of 14 pieces. Aug 8 05:07:50.814243345 (19h28m25.329345924s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.814400035 (19h28m25.329502614s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: expected piece hash; context canceled; EOF Aug 8 05:07:50.814943535 (19h28m25.330046104s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:52880->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815037104 (19h28m25.330139685s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:35090->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815074873 (19h28m25.33017744s) [seg_pos={0 0}, upload=20117]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:07:50.815158138 (19h28m25.33026072s) [upload=20117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:07:50.815163707 (19h28m25.330266293s) [upload=20117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:07:50.895420076 (19h28m25.410522646s) [upload=20117]: Done uploading segments... Aug 8 05:07:50.895438293 (19h28m25.41054086s) [upload=20117]: Upload finished. err: <nil> Aug 8 05:09:01.786221211 (19h29m36.301323791s) [upload=20144]: Starting upload Aug 8 05:09:01.786280773 (19h29m36.301383344s) [upload=20144]: Done starting upload Aug 8 05:09:01.786606643 (19h29m36.301709213s) [upload=20144]: Uploading segments... Aug 8 05:09:01.786659628 (19h29m36.301762216s) [upload=20144]: Got next segment. Inline: false Aug 8 05:09:01.786720249 (19h29m36.301822836s) [upload=20144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:09:01.786725788 (19h29m36.301828371s) [upload=20144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:09:01.797061485 (19h29m36.312164075s) [upload=20144]: Next segment err: metaclient: Too Many Requests Duration: 650.440179ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 753.161121ms FAIL (2025-08-08 05:15:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 753.187069ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 188.87556ms FAIL (2025-08-08 05:15:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 188.893496ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 430.610186ms FAIL (2025-08-08 05:14:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 823884 (19h28m25.204926455s) [seg_pos={0 0}, upload=20117]: Done begin upload segment. Aug 8 05:07:50.689845352 (19h28m25.204947931s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.689876284 (19h28m25.20497887s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690299079 (19h28m25.205401961s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690414644 (19h28m25.205517222s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690692455 (19h28m25.205795037s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690753877 (19h28m25.205856454s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691078732 (19h28m25.206181312s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691274182 (19h28m25.206376764s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691315491 (19h28m25.206418073s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691540668 (19h28m25.20664325s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691764660 (19h28m25.206867235s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691991240 (19h28m25.207093819s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692208677 (19h28m25.207311255s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692483009 (19h28m25.207585591s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692631937 (19h28m25.20773452s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692956125 (19h28m25.208058727s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693275480 (19h28m25.208378064s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693517734 (19h28m25.208620316s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.796953768 (19h28m25.312056349s) [upload=20117]: Next returned nil segment Aug 8 05:07:50.796969939 (19h28m25.312072524s) [upload=20117]: Waiting for error group managing segments... Aug 8 05:07:50.807280798 (19h28m25.322383368s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.807499572 (19h28m25.32260214s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808409851 (19h28m25.323512419s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808537380 (19h28m25.32363996s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809142026 (19h28m25.324244594s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809727637 (19h28m25.324830208s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810036385 (19h28m25.325138952s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810524084 (19h28m25.325626646s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811239458 (19h28m25.326342021s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811747718 (19h28m25.326850288s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812233806 (19h28m25.327336381s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812577779 (19h28m25.327680346s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813745244 (19h28m25.328847814s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813953245 (19h28m25.329055807s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813966177 (19h28m25.329068739s) [seg_pos={0 0}, upload=20117]: Segment reached optimal threshold of 14 pieces. Aug 8 05:07:50.814243345 (19h28m25.329345924s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.814400035 (19h28m25.329502614s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: expected piece hash; context canceled; EOF Aug 8 05:07:50.814943535 (19h28m25.330046104s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:52880->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815037104 (19h28m25.330139685s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:35090->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815074873 (19h28m25.33017744s) [seg_pos={0 0}, upload=20117]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:07:50.815158138 (19h28m25.33026072s) [upload=20117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:07:50.815163707 (19h28m25.330266293s) [upload=20117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:07:50.895420076 (19h28m25.410522646s) [upload=20117]: Done uploading segments... Aug 8 05:07:50.895438293 (19h28m25.41054086s) [upload=20117]: Upload finished. err: <nil> Aug 8 05:09:01.786221211 (19h29m36.301323791s) [upload=20144]: Starting upload Aug 8 05:09:01.786280773 (19h29m36.301383344s) [upload=20144]: Done starting upload Aug 8 05:09:01.786606643 (19h29m36.301709213s) [upload=20144]: Uploading segments... Aug 8 05:09:01.786659628 (19h29m36.301762216s) [upload=20144]: Got next segment. Inline: false Aug 8 05:09:01.786720249 (19h29m36.301822836s) [upload=20144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:09:01.786725788 (19h29m36.301828371s) [upload=20144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:09:01.797061485 (19h29m36.312164075s) [upload=20144]: Next segment err: metaclient: Too Many Requests Duration: 430.639711ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 516.090606ms FAIL (2025-08-08 05:14:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 516.111041ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 424.095297ms FAIL (2025-08-08 05:14:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 823884 (19h28m25.204926455s) [seg_pos={0 0}, upload=20117]: Done begin upload segment. Aug 8 05:07:50.689845352 (19h28m25.204947931s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.689876284 (19h28m25.20497887s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690299079 (19h28m25.205401961s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690414644 (19h28m25.205517222s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690692455 (19h28m25.205795037s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690753877 (19h28m25.205856454s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691078732 (19h28m25.206181312s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691274182 (19h28m25.206376764s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691315491 (19h28m25.206418073s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691540668 (19h28m25.20664325s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691764660 (19h28m25.206867235s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691991240 (19h28m25.207093819s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692208677 (19h28m25.207311255s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692483009 (19h28m25.207585591s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692631937 (19h28m25.20773452s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692956125 (19h28m25.208058727s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693275480 (19h28m25.208378064s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693517734 (19h28m25.208620316s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.796953768 (19h28m25.312056349s) [upload=20117]: Next returned nil segment Aug 8 05:07:50.796969939 (19h28m25.312072524s) [upload=20117]: Waiting for error group managing segments... Aug 8 05:07:50.807280798 (19h28m25.322383368s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.807499572 (19h28m25.32260214s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808409851 (19h28m25.323512419s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808537380 (19h28m25.32363996s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809142026 (19h28m25.324244594s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809727637 (19h28m25.324830208s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810036385 (19h28m25.325138952s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810524084 (19h28m25.325626646s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811239458 (19h28m25.326342021s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811747718 (19h28m25.326850288s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812233806 (19h28m25.327336381s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812577779 (19h28m25.327680346s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813745244 (19h28m25.328847814s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813953245 (19h28m25.329055807s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813966177 (19h28m25.329068739s) [seg_pos={0 0}, upload=20117]: Segment reached optimal threshold of 14 pieces. Aug 8 05:07:50.814243345 (19h28m25.329345924s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.814400035 (19h28m25.329502614s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: expected piece hash; context canceled; EOF Aug 8 05:07:50.814943535 (19h28m25.330046104s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:52880->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815037104 (19h28m25.330139685s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:35090->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815074873 (19h28m25.33017744s) [seg_pos={0 0}, upload=20117]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:07:50.815158138 (19h28m25.33026072s) [upload=20117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:07:50.815163707 (19h28m25.330266293s) [upload=20117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:07:50.895420076 (19h28m25.410522646s) [upload=20117]: Done uploading segments... Aug 8 05:07:50.895438293 (19h28m25.41054086s) [upload=20117]: Upload finished. err: <nil> Aug 8 05:09:01.786221211 (19h29m36.301323791s) [upload=20144]: Starting upload Aug 8 05:09:01.786280773 (19h29m36.301383344s) [upload=20144]: Done starting upload Aug 8 05:09:01.786606643 (19h29m36.301709213s) [upload=20144]: Uploading segments... Aug 8 05:09:01.786659628 (19h29m36.301762216s) [upload=20144]: Got next segment. Inline: false Aug 8 05:09:01.786720249 (19h29m36.301822836s) [upload=20144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:09:01.786725788 (19h29m36.301828371s) [upload=20144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:09:01.797061485 (19h29m36.312164075s) [upload=20144]: Next segment err: metaclient: Too Many Requests Duration: 424.121066ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 410.881293ms FAIL (2025-08-08 05:14:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 410.907269ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 655.835981ms FAIL (2025-08-08 05:13:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 655.860319ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 178.072903ms FAIL (2025-08-08 05:13:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 178.101448ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 481.219055ms FAIL (2025-08-08 05:13:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 481.240802ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 539.558938ms FAIL (2025-08-08 05:13:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 823884 (19h28m25.204926455s) [seg_pos={0 0}, upload=20117]: Done begin upload segment. Aug 8 05:07:50.689845352 (19h28m25.204947931s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.689876284 (19h28m25.20497887s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690299079 (19h28m25.205401961s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690414644 (19h28m25.205517222s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690692455 (19h28m25.205795037s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690753877 (19h28m25.205856454s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691078732 (19h28m25.206181312s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691274182 (19h28m25.206376764s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691315491 (19h28m25.206418073s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691540668 (19h28m25.20664325s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691764660 (19h28m25.206867235s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691991240 (19h28m25.207093819s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692208677 (19h28m25.207311255s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692483009 (19h28m25.207585591s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692631937 (19h28m25.20773452s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692956125 (19h28m25.208058727s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693275480 (19h28m25.208378064s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693517734 (19h28m25.208620316s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.796953768 (19h28m25.312056349s) [upload=20117]: Next returned nil segment Aug 8 05:07:50.796969939 (19h28m25.312072524s) [upload=20117]: Waiting for error group managing segments... Aug 8 05:07:50.807280798 (19h28m25.322383368s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.807499572 (19h28m25.32260214s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808409851 (19h28m25.323512419s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808537380 (19h28m25.32363996s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809142026 (19h28m25.324244594s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809727637 (19h28m25.324830208s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810036385 (19h28m25.325138952s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810524084 (19h28m25.325626646s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811239458 (19h28m25.326342021s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811747718 (19h28m25.326850288s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812233806 (19h28m25.327336381s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812577779 (19h28m25.327680346s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813745244 (19h28m25.328847814s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813953245 (19h28m25.329055807s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813966177 (19h28m25.329068739s) [seg_pos={0 0}, upload=20117]: Segment reached optimal threshold of 14 pieces. Aug 8 05:07:50.814243345 (19h28m25.329345924s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.814400035 (19h28m25.329502614s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: expected piece hash; context canceled; EOF Aug 8 05:07:50.814943535 (19h28m25.330046104s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:52880->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815037104 (19h28m25.330139685s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:35090->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815074873 (19h28m25.33017744s) [seg_pos={0 0}, upload=20117]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:07:50.815158138 (19h28m25.33026072s) [upload=20117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:07:50.815163707 (19h28m25.330266293s) [upload=20117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:07:50.895420076 (19h28m25.410522646s) [upload=20117]: Done uploading segments... Aug 8 05:07:50.895438293 (19h28m25.41054086s) [upload=20117]: Upload finished. err: <nil> Aug 8 05:09:01.786221211 (19h29m36.301323791s) [upload=20144]: Starting upload Aug 8 05:09:01.786280773 (19h29m36.301383344s) [upload=20144]: Done starting upload Aug 8 05:09:01.786606643 (19h29m36.301709213s) [upload=20144]: Uploading segments... Aug 8 05:09:01.786659628 (19h29m36.301762216s) [upload=20144]: Got next segment. Inline: false Aug 8 05:09:01.786720249 (19h29m36.301822836s) [upload=20144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:09:01.786725788 (19h29m36.301828371s) [upload=20144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:09:01.797061485 (19h29m36.312164075s) [upload=20144]: Next segment err: metaclient: Too Many Requests Duration: 539.587497ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 513.71733ms FAIL (2025-08-08 05:13:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 513.745827ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 190.713574ms FAIL (2025-08-08 05:12:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 190.743008ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 427.892115ms FAIL (2025-08-08 05:12:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 823884 (19h28m25.204926455s) [seg_pos={0 0}, upload=20117]: Done begin upload segment. Aug 8 05:07:50.689845352 (19h28m25.204947931s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.689876284 (19h28m25.20497887s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690299079 (19h28m25.205401961s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690414644 (19h28m25.205517222s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690692455 (19h28m25.205795037s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690753877 (19h28m25.205856454s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691078732 (19h28m25.206181312s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691274182 (19h28m25.206376764s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691315491 (19h28m25.206418073s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691540668 (19h28m25.20664325s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691764660 (19h28m25.206867235s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691991240 (19h28m25.207093819s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692208677 (19h28m25.207311255s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692483009 (19h28m25.207585591s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692631937 (19h28m25.20773452s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692956125 (19h28m25.208058727s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693275480 (19h28m25.208378064s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693517734 (19h28m25.208620316s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.796953768 (19h28m25.312056349s) [upload=20117]: Next returned nil segment Aug 8 05:07:50.796969939 (19h28m25.312072524s) [upload=20117]: Waiting for error group managing segments... Aug 8 05:07:50.807280798 (19h28m25.322383368s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.807499572 (19h28m25.32260214s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808409851 (19h28m25.323512419s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808537380 (19h28m25.32363996s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809142026 (19h28m25.324244594s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809727637 (19h28m25.324830208s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810036385 (19h28m25.325138952s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810524084 (19h28m25.325626646s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811239458 (19h28m25.326342021s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811747718 (19h28m25.326850288s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812233806 (19h28m25.327336381s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812577779 (19h28m25.327680346s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813745244 (19h28m25.328847814s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813953245 (19h28m25.329055807s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813966177 (19h28m25.329068739s) [seg_pos={0 0}, upload=20117]: Segment reached optimal threshold of 14 pieces. Aug 8 05:07:50.814243345 (19h28m25.329345924s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.814400035 (19h28m25.329502614s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: expected piece hash; context canceled; EOF Aug 8 05:07:50.814943535 (19h28m25.330046104s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:52880->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815037104 (19h28m25.330139685s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:35090->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815074873 (19h28m25.33017744s) [seg_pos={0 0}, upload=20117]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:07:50.815158138 (19h28m25.33026072s) [upload=20117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:07:50.815163707 (19h28m25.330266293s) [upload=20117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:07:50.895420076 (19h28m25.410522646s) [upload=20117]: Done uploading segments... Aug 8 05:07:50.895438293 (19h28m25.41054086s) [upload=20117]: Upload finished. err: <nil> Aug 8 05:09:01.786221211 (19h29m36.301323791s) [upload=20144]: Starting upload Aug 8 05:09:01.786280773 (19h29m36.301383344s) [upload=20144]: Done starting upload Aug 8 05:09:01.786606643 (19h29m36.301709213s) [upload=20144]: Uploading segments... Aug 8 05:09:01.786659628 (19h29m36.301762216s) [upload=20144]: Got next segment. Inline: false Aug 8 05:09:01.786720249 (19h29m36.301822836s) [upload=20144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:09:01.786725788 (19h29m36.301828371s) [upload=20144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:09:01.797061485 (19h29m36.312164075s) [upload=20144]: Next segment err: metaclient: Too Many Requests Duration: 427.918819ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 521.46172ms FAIL (2025-08-08 05:12:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 521.483352ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 414.294797ms FAIL (2025-08-08 05:12:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 823884 (19h28m25.204926455s) [seg_pos={0 0}, upload=20117]: Done begin upload segment. Aug 8 05:07:50.689845352 (19h28m25.204947931s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.689876284 (19h28m25.20497887s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690299079 (19h28m25.205401961s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690414644 (19h28m25.205517222s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690692455 (19h28m25.205795037s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690753877 (19h28m25.205856454s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691078732 (19h28m25.206181312s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691274182 (19h28m25.206376764s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691315491 (19h28m25.206418073s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691540668 (19h28m25.20664325s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691764660 (19h28m25.206867235s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691991240 (19h28m25.207093819s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692208677 (19h28m25.207311255s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692483009 (19h28m25.207585591s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692631937 (19h28m25.20773452s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692956125 (19h28m25.208058727s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693275480 (19h28m25.208378064s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693517734 (19h28m25.208620316s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.796953768 (19h28m25.312056349s) [upload=20117]: Next returned nil segment Aug 8 05:07:50.796969939 (19h28m25.312072524s) [upload=20117]: Waiting for error group managing segments... Aug 8 05:07:50.807280798 (19h28m25.322383368s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.807499572 (19h28m25.32260214s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808409851 (19h28m25.323512419s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808537380 (19h28m25.32363996s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809142026 (19h28m25.324244594s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809727637 (19h28m25.324830208s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810036385 (19h28m25.325138952s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810524084 (19h28m25.325626646s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811239458 (19h28m25.326342021s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811747718 (19h28m25.326850288s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812233806 (19h28m25.327336381s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812577779 (19h28m25.327680346s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813745244 (19h28m25.328847814s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813953245 (19h28m25.329055807s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813966177 (19h28m25.329068739s) [seg_pos={0 0}, upload=20117]: Segment reached optimal threshold of 14 pieces. Aug 8 05:07:50.814243345 (19h28m25.329345924s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.814400035 (19h28m25.329502614s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: expected piece hash; context canceled; EOF Aug 8 05:07:50.814943535 (19h28m25.330046104s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:52880->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815037104 (19h28m25.330139685s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:35090->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815074873 (19h28m25.33017744s) [seg_pos={0 0}, upload=20117]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:07:50.815158138 (19h28m25.33026072s) [upload=20117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:07:50.815163707 (19h28m25.330266293s) [upload=20117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:07:50.895420076 (19h28m25.410522646s) [upload=20117]: Done uploading segments... Aug 8 05:07:50.895438293 (19h28m25.41054086s) [upload=20117]: Upload finished. err: <nil> Aug 8 05:09:01.786221211 (19h29m36.301323791s) [upload=20144]: Starting upload Aug 8 05:09:01.786280773 (19h29m36.301383344s) [upload=20144]: Done starting upload Aug 8 05:09:01.786606643 (19h29m36.301709213s) [upload=20144]: Uploading segments... Aug 8 05:09:01.786659628 (19h29m36.301762216s) [upload=20144]: Got next segment. Inline: false Aug 8 05:09:01.786720249 (19h29m36.301822836s) [upload=20144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:09:01.786725788 (19h29m36.301828371s) [upload=20144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:09:01.797061485 (19h29m36.312164075s) [upload=20144]: Next segment err: metaclient: Too Many Requests Duration: 414.329541ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 194.96255ms FAIL (2025-08-08 05:12:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 194.996374ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 488.816178ms FAIL (2025-08-08 05:11:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 488.911789ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 180.776557ms FAIL (2025-08-08 05:11:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 180.821092ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 270.454742ms FAIL (2025-08-08 05:11:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 270.485625ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 413.681445ms FAIL (2025-08-08 05:11:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 823884 (19h28m25.204926455s) [seg_pos={0 0}, upload=20117]: Done begin upload segment. Aug 8 05:07:50.689845352 (19h28m25.204947931s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.689876284 (19h28m25.20497887s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690299079 (19h28m25.205401961s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690414644 (19h28m25.205517222s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690692455 (19h28m25.205795037s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690753877 (19h28m25.205856454s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691078732 (19h28m25.206181312s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691274182 (19h28m25.206376764s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691315491 (19h28m25.206418073s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691540668 (19h28m25.20664325s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691764660 (19h28m25.206867235s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691991240 (19h28m25.207093819s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692208677 (19h28m25.207311255s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692483009 (19h28m25.207585591s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692631937 (19h28m25.20773452s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692956125 (19h28m25.208058727s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693275480 (19h28m25.208378064s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693517734 (19h28m25.208620316s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.796953768 (19h28m25.312056349s) [upload=20117]: Next returned nil segment Aug 8 05:07:50.796969939 (19h28m25.312072524s) [upload=20117]: Waiting for error group managing segments... Aug 8 05:07:50.807280798 (19h28m25.322383368s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.807499572 (19h28m25.32260214s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808409851 (19h28m25.323512419s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808537380 (19h28m25.32363996s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809142026 (19h28m25.324244594s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809727637 (19h28m25.324830208s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810036385 (19h28m25.325138952s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810524084 (19h28m25.325626646s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811239458 (19h28m25.326342021s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811747718 (19h28m25.326850288s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812233806 (19h28m25.327336381s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812577779 (19h28m25.327680346s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813745244 (19h28m25.328847814s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813953245 (19h28m25.329055807s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813966177 (19h28m25.329068739s) [seg_pos={0 0}, upload=20117]: Segment reached optimal threshold of 14 pieces. Aug 8 05:07:50.814243345 (19h28m25.329345924s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.814400035 (19h28m25.329502614s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: expected piece hash; context canceled; EOF Aug 8 05:07:50.814943535 (19h28m25.330046104s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:52880->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815037104 (19h28m25.330139685s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:35090->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815074873 (19h28m25.33017744s) [seg_pos={0 0}, upload=20117]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:07:50.815158138 (19h28m25.33026072s) [upload=20117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:07:50.815163707 (19h28m25.330266293s) [upload=20117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:07:50.895420076 (19h28m25.410522646s) [upload=20117]: Done uploading segments... Aug 8 05:07:50.895438293 (19h28m25.41054086s) [upload=20117]: Upload finished. err: <nil> Aug 8 05:09:01.786221211 (19h29m36.301323791s) [upload=20144]: Starting upload Aug 8 05:09:01.786280773 (19h29m36.301383344s) [upload=20144]: Done starting upload Aug 8 05:09:01.786606643 (19h29m36.301709213s) [upload=20144]: Uploading segments... Aug 8 05:09:01.786659628 (19h29m36.301762216s) [upload=20144]: Got next segment. Inline: false Aug 8 05:09:01.786720249 (19h29m36.301822836s) [upload=20144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:09:01.786725788 (19h29m36.301828371s) [upload=20144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:09:01.797061485 (19h29m36.312164075s) [upload=20144]: Next segment err: metaclient: Too Many Requests Duration: 413.710573ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 483.836765ms FAIL (2025-08-08 05:10:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.861517ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 428.100721ms FAIL (2025-08-08 05:10:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 823884 (19h28m25.204926455s) [seg_pos={0 0}, upload=20117]: Done begin upload segment. Aug 8 05:07:50.689845352 (19h28m25.204947931s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.689876284 (19h28m25.20497887s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690299079 (19h28m25.205401961s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690414644 (19h28m25.205517222s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690692455 (19h28m25.205795037s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690753877 (19h28m25.205856454s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691078732 (19h28m25.206181312s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691274182 (19h28m25.206376764s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691315491 (19h28m25.206418073s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691540668 (19h28m25.20664325s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691764660 (19h28m25.206867235s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691991240 (19h28m25.207093819s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692208677 (19h28m25.207311255s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692483009 (19h28m25.207585591s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692631937 (19h28m25.20773452s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692956125 (19h28m25.208058727s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693275480 (19h28m25.208378064s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693517734 (19h28m25.208620316s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.796953768 (19h28m25.312056349s) [upload=20117]: Next returned nil segment Aug 8 05:07:50.796969939 (19h28m25.312072524s) [upload=20117]: Waiting for error group managing segments... Aug 8 05:07:50.807280798 (19h28m25.322383368s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.807499572 (19h28m25.32260214s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808409851 (19h28m25.323512419s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808537380 (19h28m25.32363996s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809142026 (19h28m25.324244594s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809727637 (19h28m25.324830208s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810036385 (19h28m25.325138952s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810524084 (19h28m25.325626646s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811239458 (19h28m25.326342021s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811747718 (19h28m25.326850288s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812233806 (19h28m25.327336381s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812577779 (19h28m25.327680346s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813745244 (19h28m25.328847814s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813953245 (19h28m25.329055807s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813966177 (19h28m25.329068739s) [seg_pos={0 0}, upload=20117]: Segment reached optimal threshold of 14 pieces. Aug 8 05:07:50.814243345 (19h28m25.329345924s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.814400035 (19h28m25.329502614s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: expected piece hash; context canceled; EOF Aug 8 05:07:50.814943535 (19h28m25.330046104s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:52880->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815037104 (19h28m25.330139685s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:35090->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815074873 (19h28m25.33017744s) [seg_pos={0 0}, upload=20117]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:07:50.815158138 (19h28m25.33026072s) [upload=20117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:07:50.815163707 (19h28m25.330266293s) [upload=20117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:07:50.895420076 (19h28m25.410522646s) [upload=20117]: Done uploading segments... Aug 8 05:07:50.895438293 (19h28m25.41054086s) [upload=20117]: Upload finished. err: <nil> Aug 8 05:09:01.786221211 (19h29m36.301323791s) [upload=20144]: Starting upload Aug 8 05:09:01.786280773 (19h29m36.301383344s) [upload=20144]: Done starting upload Aug 8 05:09:01.786606643 (19h29m36.301709213s) [upload=20144]: Uploading segments... Aug 8 05:09:01.786659628 (19h29m36.301762216s) [upload=20144]: Got next segment. Inline: false Aug 8 05:09:01.786720249 (19h29m36.301822836s) [upload=20144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:09:01.786725788 (19h29m36.301828371s) [upload=20144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:09:01.797061485 (19h29m36.312164075s) [upload=20144]: Next segment err: metaclient: Too Many Requests Duration: 428.126164ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 429.052261ms FAIL (2025-08-08 05:10:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 823884 (19h28m25.204926455s) [seg_pos={0 0}, upload=20117]: Done begin upload segment. Aug 8 05:07:50.689845352 (19h28m25.204947931s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.689876284 (19h28m25.20497887s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690299079 (19h28m25.205401961s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690414644 (19h28m25.205517222s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690692455 (19h28m25.205795037s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690753877 (19h28m25.205856454s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691078732 (19h28m25.206181312s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691274182 (19h28m25.206376764s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691315491 (19h28m25.206418073s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691540668 (19h28m25.20664325s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691764660 (19h28m25.206867235s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691991240 (19h28m25.207093819s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692208677 (19h28m25.207311255s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692483009 (19h28m25.207585591s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692631937 (19h28m25.20773452s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692956125 (19h28m25.208058727s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693275480 (19h28m25.208378064s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693517734 (19h28m25.208620316s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.796953768 (19h28m25.312056349s) [upload=20117]: Next returned nil segment Aug 8 05:07:50.796969939 (19h28m25.312072524s) [upload=20117]: Waiting for error group managing segments... Aug 8 05:07:50.807280798 (19h28m25.322383368s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.807499572 (19h28m25.32260214s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808409851 (19h28m25.323512419s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808537380 (19h28m25.32363996s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809142026 (19h28m25.324244594s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809727637 (19h28m25.324830208s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810036385 (19h28m25.325138952s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810524084 (19h28m25.325626646s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811239458 (19h28m25.326342021s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811747718 (19h28m25.326850288s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812233806 (19h28m25.327336381s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812577779 (19h28m25.327680346s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813745244 (19h28m25.328847814s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813953245 (19h28m25.329055807s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813966177 (19h28m25.329068739s) [seg_pos={0 0}, upload=20117]: Segment reached optimal threshold of 14 pieces. Aug 8 05:07:50.814243345 (19h28m25.329345924s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.814400035 (19h28m25.329502614s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: expected piece hash; context canceled; EOF Aug 8 05:07:50.814943535 (19h28m25.330046104s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:52880->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815037104 (19h28m25.330139685s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:35090->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815074873 (19h28m25.33017744s) [seg_pos={0 0}, upload=20117]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:07:50.815158138 (19h28m25.33026072s) [upload=20117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:07:50.815163707 (19h28m25.330266293s) [upload=20117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:07:50.895420076 (19h28m25.410522646s) [upload=20117]: Done uploading segments... Aug 8 05:07:50.895438293 (19h28m25.41054086s) [upload=20117]: Upload finished. err: <nil> Aug 8 05:09:01.786221211 (19h29m36.301323791s) [upload=20144]: Starting upload Aug 8 05:09:01.786280773 (19h29m36.301383344s) [upload=20144]: Done starting upload Aug 8 05:09:01.786606643 (19h29m36.301709213s) [upload=20144]: Uploading segments... Aug 8 05:09:01.786659628 (19h29m36.301762216s) [upload=20144]: Got next segment. Inline: false Aug 8 05:09:01.786720249 (19h29m36.301822836s) [upload=20144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:09:01.786725788 (19h29m36.301828371s) [upload=20144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:09:01.797061485 (19h29m36.312164075s) [upload=20144]: Next segment err: metaclient: Too Many Requests Duration: 429.10448ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 222.656712ms FAIL (2025-08-08 05:10:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== uestItem_ObjectCommit Aug 8 04:42:16.180307652 (19h2m50.695410219s) [upload=19671]: Done uploading segments... Aug 8 04:42:16.180326351 (19h2m50.695428913s) [upload=19671]: Upload finished. err: <nil> Aug 8 04:42:41.083579476 (19h3m15.598682039s) [upload=19678]: Starting upload Aug 8 04:42:41.083658806 (19h3m15.598761384s) [upload=19678]: Done starting upload Aug 8 04:42:41.083736126 (19h3m15.598838696s) [upload=19678]: Uploading segments... Aug 8 04:42:41.083798735 (19h3m15.59890132s) [upload=19678]: Got next segment. Inline: true Aug 8 04:42:41.083805351 (19h3m15.598907934s) [upload=19678]: Waiting for error group managing segments... Aug 8 04:42:41.083894019 (19h3m15.598996595s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:42:41.083899702 (19h3m15.599002278s) [upload=19678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:42:41.083903543 (19h3m15.59900612s) [upload=19678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:42:41.147259315 (19h3m15.662361882s) [upload=19678]: Done uploading segments... Aug 8 04:42:41.147276213 (19h3m15.66237879s) [upload=19678]: Upload finished. err: <nil> Aug 8 04:43:54.314265905 (19h4m28.829368481s) [upload=19696]: Starting upload Aug 8 04:43:54.314342572 (19h4m28.829445149s) [upload=19696]: Done starting upload Aug 8 04:43:54.314412786 (19h4m28.829515355s) [upload=19696]: Uploading segments... Aug 8 04:43:54.314461737 (19h4m28.829564323s) [upload=19696]: Got next segment. Inline: true Aug 8 04:43:54.314468588 (19h4m28.829571166s) [upload=19696]: Waiting for error group managing segments... Aug 8 04:43:54.314513835 (19h4m28.829616411s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:43:54.314518357 (19h4m28.829620933s) [upload=19696]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:43:54.314521699 (19h4m28.829624275s) [upload=19696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:43:54.374490000 (19h4m28.889592577s) [upload=19696]: Done uploading segments... Aug 8 04:43:54.374507640 (19h4m28.889610205s) [upload=19696]: Upload finished. err: <nil> Aug 8 04:44:37.820610246 (19h5m12.335712815s) [upload=19708]: Starting upload Aug 8 04:44:37.820681721 (19h5m12.335784288s) [upload=19708]: Done starting upload Aug 8 04:44:37.820750998 (19h5m12.335853567s) [upload=19708]: Uploading segments... Aug 8 04:44:37.820847572 (19h5m12.335950158s) [upload=19708]: Got next segment. Inline: true Aug 8 04:44:37.820871197 (19h5m12.335973778s) [upload=19708]: Waiting for error group managing segments... Aug 8 04:44:37.820923988 (19h5m12.336026577s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:44:37.820930149 (19h5m12.33603273s) [upload=19708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:44:37.820950094 (19h5m12.336052671s) [upload=19708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:44:37.891304392 (19h5m12.406406974s) [upload=19708]: Done uploading segments... Aug 8 04:44:37.891324719 (19h5m12.406427302s) [upload=19708]: Upload finished. err: <nil> Aug 8 04:45:49.374287050 (19h6m23.889389635s) [upload=19727]: Starting upload Aug 8 04:45:49.374385480 (19h6m23.889488062s) [upload=19727]: Done starting upload Aug 8 04:45:49.374485186 (19h6m23.889587763s) [upload=19727]: Uploading segments... Aug 8 04:45:49.374553702 (19h6m23.889656288s) [upload=19727]: Got next segment. Inline: true Aug 8 04:45:49.374559271 (19h6m23.889661849s) [upload=19727]: Waiting for error group managing segments... Aug 8 04:45:49.374607801 (19h6m23.889710384s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:45:49.374613021 (19h6m23.889715604s) [upload=19727]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:45:49.374616970 (19h6m23.889719552s) [upload=19727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:45:49.440532379 (19h6m23.955634956s) [upload=19727]: Done uploading segments... Aug 8 04:45:49.440553942 (19h6m23.955656506s) [upload=19727]: Upload finished. err: <nil> Aug 8 04:46:22.148691181 (19h6m56.663793758s) [upload=19736]: Starting upload Aug 8 04:46:22.148731264 (19h6m56.66383385s) [upload=19736]: Done starting upload Aug 8 04:46:22.148779839 (19h6m56.663882416s) [upload=19736]: Uploading segments... Aug 8 04:46:22.148815522 (19h6m56.663918108s) [upload=19736]: Got next segment. Inline: true Aug 8 04:46:22.148819611 (19h6m56.663922186s) [upload=19736]: Waiting for error group managing segments... Aug 8 04:46:22.148853347 (19h6m56.663955925s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:22.148856905 (19h6m56.663959481s) [upload=19736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:22.148859219 (19h6m56.663961798s) [upload=19736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:22.224160590 (19h6m56.739263173s) [upload=19736]: Done uploading segments... Aug 8 04:46:22.224183380 (19h6m56.739285958s) [upload=19736]: Upload finished. err: <nil> Aug 8 04:46:57.599668727 (19h7m32.114771304s) [upload=19750]: Starting upload Aug 8 04:46:57.599733716 (19h7m32.114836292s) [upload=19750]: Done starting upload Aug 8 04:46:57.599831148 (19h7m32.114933716s) [upload=19750]: Uploading segments... Aug 8 04:46:57.599898194 (19h7m32.115000774s) [upload=19750]: Got next segment. Inline: true Aug 8 04:46:57.599902995 (19h7m32.115005571s) [upload=19750]: Waiting for error group managing segments... Aug 8 04:46:57.599955730 (19h7m32.115058297s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:46:57.599960210 (19h7m32.115062786s) [upload=19750]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:46:57.599963339 (19h7m32.115065916s) [upload=19750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:46:57.661947764 (19h7m32.17705034s) [upload=19750]: Done uploading segments... Aug 8 04:46:57.661964723 (19h7m32.1770673s) [upload=19750]: Upload finished. err: <nil> Aug 8 04:48:07.498218136 (19h8m42.013320702s) [upload=19766]: Starting upload Aug 8 04:48:07.498272608 (19h8m42.013375174s) [upload=19766]: Done starting upload Aug 8 04:48:07.498340076 (19h8m42.013442642s) [upload=19766]: Uploading segments... Aug 8 04:48:07.498377092 (19h8m42.013479678s) [upload=19766]: Got next segment. Inline: true Aug 8 04:48:07.498381514 (19h8m42.013484091s) [upload=19766]: Waiting for error group managing segments... Aug 8 04:48:07.498422115 (19h8m42.013524681s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:07.498426401 (19h8m42.013528979s) [upload=19766]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:07.498429474 (19h8m42.01353205s) [upload=19766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:07.573834459 (19h8m42.088937036s) [upload=19766]: Done uploading segments... Aug 8 04:48:07.573849292 (19h8m42.088951861s) [upload=19766]: Upload finished. err: <nil> Aug 8 04:48:24.131081943 (19h8m58.646184521s) [upload=19773]: Starting upload Aug 8 04:48:24.131135304 (19h8m58.646237872s) [upload=19773]: Done starting upload Aug 8 04:48:24.131203832 (19h8m58.646306401s) [upload=19773]: Uploading segments... Aug 8 04:48:24.131242000 (19h8m58.646344585s) [upload=19773]: Got next segment. Inline: true Aug 8 04:48:24.131246356 (19h8m58.646348932s) [upload=19773]: Waiting for error group managing segments... Aug 8 04:48:24.131298903 (19h8m58.646401482s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:48:24.131304610 (19h8m58.646407189s) [upload=19773]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:48:24.131307703 (19h8m58.646410279s) [upload=19773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:48:24.189165262 (19h8m58.704267839s) [upload=19773]: Done uploading segments... Aug 8 04:48:24.189182600 (19h8m58.704285176s) [upload=19773]: Upload finished. err: <nil> Aug 8 04:49:30.179743426 (19h10m4.69484601s) [upload=19792]: Starting upload Aug 8 04:49:30.179830624 (19h10m4.694933205s) [upload=19792]: Done starting upload Aug 8 04:49:30.179932276 (19h10m4.695034844s) [upload=19792]: Uploading segments... Aug 8 04:49:30.180011264 (19h10m4.69511385s) [upload=19792]: Got next segment. Inline: true Aug 8 04:49:30.180031508 (19h10m4.695134094s) [upload=19792]: Waiting for error group managing segments... Aug 8 04:49:30.180083780 (19h10m4.695186362s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:30.180089451 (19h10m4.695192035s) [upload=19792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:30.180107234 (19h10m4.695209811s) [upload=19792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:30.255066803 (19h10m4.770169372s) [upload=19792]: Done uploading segments... Aug 8 04:49:30.255097834 (19h10m4.770200412s) [upload=19792]: Upload finished. err: <nil> Aug 8 04:49:49.722537438 (19h10m24.237640014s) [upload=19800]: Starting upload Aug 8 04:49:49.722597342 (19h10m24.237699917s) [upload=19800]: Done starting upload Aug 8 04:49:49.722664052 (19h10m24.237766615s) [upload=19800]: Uploading segments... Aug 8 04:49:49.722715327 (19h10m24.23781791s) [upload=19800]: Got next segment. Inline: true Aug 8 04:49:49.722719940 (19h10m24.237822515s) [upload=19800]: Waiting for error group managing segments... Aug 8 04:49:49.722753934 (19h10m24.23785651s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 04:49:49.722758369 (19h10m24.237860931s) [upload=19800]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 04:49:49.722815656 (19h10m24.237918234s) [upload=19800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 04:49:49.824454883 (19h10m24.339557452s) [upload=19800]: Done uploading segments... Aug 8 04:49:49.824470680 (19h10m24.339573247s) [upload=19800]: Upload finished. err: <nil> Duration: 222.709392ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 511.342454ms FAIL (2025-08-08 05:09:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 511.410975ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.022462874s FAIL (2025-08-08 05:09:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 823884 (19h28m25.204926455s) [seg_pos={0 0}, upload=20117]: Done begin upload segment. Aug 8 05:07:50.689845352 (19h28m25.204947931s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.689876284 (19h28m25.20497887s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690299079 (19h28m25.205401961s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690414644 (19h28m25.205517222s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690692455 (19h28m25.205795037s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.690753877 (19h28m25.205856454s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691078732 (19h28m25.206181312s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691274182 (19h28m25.206376764s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691315491 (19h28m25.206418073s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691540668 (19h28m25.20664325s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691764660 (19h28m25.206867235s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.691991240 (19h28m25.207093819s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692208677 (19h28m25.207311255s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692483009 (19h28m25.207585591s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692631937 (19h28m25.20773452s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.692956125 (19h28m25.208058727s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693275480 (19h28m25.208378064s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.693517734 (19h28m25.208620316s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Uploading piece... Aug 8 05:07:50.796953768 (19h28m25.312056349s) [upload=20117]: Next returned nil segment Aug 8 05:07:50.796969939 (19h28m25.312072524s) [upload=20117]: Waiting for error group managing segments... Aug 8 05:07:50.807280798 (19h28m25.322383368s) [noise=true , address=104.36.82.44:4904 , piece_id=OBWUIWKTMV7WNHLWYMVWNCEDYKVQRTHVSD2QXBIW4Y7R2LHQEX2Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.807499572 (19h28m25.32260214s) [noise=true , address=104.36.82.62:4409 , piece_id=6ETQE3PYGLP2LUKEVYQT5GUYOU4HOXSFSTEMWBE5O4OQQPZ45ROQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808409851 (19h28m25.323512419s) [noise=true , address=104.36.82.60:4360 , piece_id=UIOWINYLBEQPYZA4VZIROITCEQNL3Y7VBC5DD6RFUOHYJRRAWGXQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.808537380 (19h28m25.32363996s) [noise=true , address=104.36.82.43:4839 , piece_id=D3LL37FREL2NLARF7ZNDMIIM5MTIYWT6HKHBGIIJ4WAU3JTSXIMA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809142026 (19h28m25.324244594s) [noise=true , address=104.36.82.59:4329 , piece_id=TKQYQW3K6TAPCKEMQEBLAMPDWM4ZTKUW2DVQLGO5XAADU5ZVM7MQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.809727637 (19h28m25.324830208s) [noise=true , address=104.36.82.23:4479 , piece_id=OWQGKE2OPGKDKME3ZM5QP7ICZI7DGK64JGDDYZ2Y2XJ7I2SYDDHA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810036385 (19h28m25.325138952s) [noise=true , address=104.36.82.20:4451 , piece_id=CJ57ABBCVOSQPID254BOCGQYV5TCKBX75X3I4TTJPBPDAB3SRGZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.810524084 (19h28m25.325626646s) [noise=true , address=104.36.82.46:4503 , piece_id=JZMUA6HDO3UKWDD4OLCN4VEDYOVNRCR7YI2FSFHOGCVFYGTRAVVQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811239458 (19h28m25.326342021s) [noise=true , address=104.36.82.18:4398 , piece_id=RHEHRWJAGRBFJ7FCBK2AP2QKHRQ6T3SCAU6P6N2676XRX3EIQFQQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.811747718 (19h28m25.326850288s) [noise=true , address=104.36.82.47:4530 , piece_id=OKRF4KVXISXVUTIV4BQGBJOQ62UJK2MA32UI6XM5EYC2Y7PJ4PZQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812233806 (19h28m25.327336381s) [noise=true , address=104.36.82.58:4273 , piece_id=ZGYB4SSWJNCMKAARDCNSEQMFAQJN7API2HERTCUZP5UYARRSSFPA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.812577779 (19h28m25.327680346s) [noise=true , address=104.36.82.12:4163 , piece_id=TLKBIUZ3LCQWCRMHRDZRRI62UKU7ACUDS3YYEOU3K6DASXRCS3VA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813745244 (19h28m25.328847814s) [noise=true , address=104.36.82.38:4696 , piece_id=JA4R7FCB2U243LV5LFX4TVIKTKQSXJLM3PMPPCPW45MO75I25XTQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813953245 (19h28m25.329055807s) [noise=true , address=104.36.82.37:4681 , piece_id=7ZFQFX2Q6C4YZNLIORFAF4QVL7H4QCBZHDVJI2TR2ZRY4HI67TQA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.813966177 (19h28m25.329068739s) [seg_pos={0 0}, upload=20117]: Segment reached optimal threshold of 14 pieces. Aug 8 05:07:50.814243345 (19h28m25.329345924s) [noise=true , address=104.36.82.56:4242 , piece_id=VF4TASC7USPMTSXWGYQNNTLDPJE2YXDJBAPGKSA3SYDAPGZKZYGA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: <nil> Aug 8 05:07:50.814400035 (19h28m25.329502614s) [noise=true , address=104.36.82.4:4811 , piece_id=BITGHQSFC2QWI3MT6WW4RQBVEZLGCWZOLEAZDQT5HKKEIS7BQVGQ, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: expected piece hash; context canceled; EOF Aug 8 05:07:50.814943535 (19h28m25.330046104s) [noise=true , address=64.49.13.170:4725 , piece_id=MAWII32TCXPSBBIZ6N3UH2SI6GQXXDY5YYMQXEKAAGBXZ3P7EZXA, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:52880->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815037104 (19h28m25.330139685s) [noise=true , address=64.49.13.138:4486 , piece_id=QQQFYIR7LMDJVBCPKRKDYOS5K6PT7V47427ITU33TBHHQG4TOH7Q, seg_pos={0 0}, upload=20117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:35090->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 05:07:50.815074873 (19h28m25.33017744s) [seg_pos={0 0}, upload=20117]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Aug 8 05:07:50.815158138 (19h28m25.33026072s) [upload=20117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 05:07:50.815163707 (19h28m25.330266293s) [upload=20117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 05:07:50.895420076 (19h28m25.410522646s) [upload=20117]: Done uploading segments... Aug 8 05:07:50.895438293 (19h28m25.41054086s) [upload=20117]: Upload finished. err: <nil> Aug 8 05:09:01.786221211 (19h29m36.301323791s) [upload=20144]: Starting upload Aug 8 05:09:01.786280773 (19h29m36.301383344s) [upload=20144]: Done starting upload Aug 8 05:09:01.786606643 (19h29m36.301709213s) [upload=20144]: Uploading segments... Aug 8 05:09:01.786659628 (19h29m36.301762216s) [upload=20144]: Got next segment. Inline: false Aug 8 05:09:01.786720249 (19h29m36.301822836s) [upload=20144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 05:09:01.786725788 (19h29m36.301828371s) [upload=20144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 05:09:01.797061485 (19h29m36.312164075s) [upload=20144]: Next segment err: metaclient: Too Many Requests Duration: 1.022491402s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)