Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 426.188252ms FAIL (2025-07-04 19:19:55 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 426.214899ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 451.662343ms FAIL (2025-07-04 19:19:27 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 451.770361ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 459.164711ms FAIL (2025-07-04 19:19:11 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 459.196832ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 542.410215ms FAIL (2025-07-04 19:18:26 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 542.442546ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 492.40077ms FAIL (2025-07-04 19:17:22 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 492.451342ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 427.188403ms FAIL (2025-07-04 19:16:12 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 427.212861ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 543.761541ms FAIL (2025-07-04 19:15:53 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 543.789754ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 442.612858ms FAIL (2025-07-04 19:14:39 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 442.643451ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 436.901227ms FAIL (2025-07-04 19:13:43 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 436.948674ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 529.307582ms FAIL (2025-07-04 19:12:47 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 529.334876ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 472.108889ms FAIL (2025-07-04 19:12:03 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 472.133583ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 454.041435ms FAIL (2025-07-04 19:11:06 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 454.069388ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 472.56199ms FAIL (2025-07-04 19:10:10 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 472.591359ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 401.57765ms FAIL (2025-07-04 19:09:25 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 401.606942ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 448.144002ms FAIL (2025-07-04 19:08:12 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 448.181008ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 445.544502ms FAIL (2025-07-04 19:06:58 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 445.581247ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 452.237667ms FAIL (2025-07-04 19:06:08 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 452.265589ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 442.978495ms FAIL (2025-07-04 19:05:28 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 443.008861ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 607.333838ms FAIL (2025-07-04 19:04:43 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 607.367243ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 457.598357ms FAIL (2025-07-04 19:04:09 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 457.630802ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 541.842333ms FAIL (2025-07-04 19:03:36 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 541.891035ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000841213s FAIL (2025-07-04 19:02:51 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ent: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jul 4 19:00:31.104437105 (273h48m58.030731786s) [noise=true , address=104.36.82.6:4876 , piece_id=IO2KOOKKFG6UX5AD6UVT3G5LVSM5WQCZGBE4VSDVNE7QSPEICSUQ, seg_pos={0 0}, upload=219282]: Done uploading piece. err: <nil> Jul 4 19:00:31.104527728 (273h48m58.030822419s) [seg_pos={0 0}, upload=219282]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 4 19:00:31.104648944 (273h48m58.030943625s) [upload=219282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:00:31.104655455 (273h48m58.030950142s) [upload=219282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:00:31.162750349 (273h48m58.089045031s) [upload=219282]: Done uploading segments... Jul 4 19:00:31.162773340 (273h48m58.089068016s) [upload=219282]: Upload finished. err: <nil> Jul 4 19:01:21.112342853 (273h49m48.038637539s) [upload=219293]: Starting upload Jul 4 19:01:21.112455651 (273h49m48.038750342s) [upload=219293]: Done starting upload Jul 4 19:01:21.112553742 (273h49m48.038848432s) [upload=219293]: Uploading segments... Jul 4 19:01:21.112736460 (273h49m48.039031159s) [upload=219293]: Got next segment. Inline: false Jul 4 19:01:21.112791315 (273h49m48.039086005s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 19:01:21.112797383 (273h49m48.039092074s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 19:01:21.143531966 (273h49m48.069826653s) [seg_pos={0 0}, upload=219293]: Begin upload segment... Jul 4 19:01:21.143608120 (273h49m48.069902811s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144145375 (273h49m48.070440063s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.144692717 (273h49m48.070987395s) [seg_pos={0 0}, upload=219293]: Done begin upload segment. Jul 4 19:01:21.144743385 (273h49m48.071038077s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145101631 (273h49m48.071396318s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145392058 (273h49m48.071686744s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145625172 (273h49m48.071919863s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.145946419 (273h49m48.072241116s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.146359714 (273h49m48.07265441s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147285866 (273h49m48.073580563s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.147769098 (273h49m48.074063799s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149434914 (273h49m48.075729603s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.149821203 (273h49m48.076115891s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.150943260 (273h49m48.077237949s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.151677976 (273h49m48.077972668s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152030529 (273h49m48.078325219s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.152498377 (273h49m48.078793073s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Uploading piece... Jul 4 19:01:21.270073650 (273h49m48.196368338s) [upload=219293]: Next returned nil segment Jul 4 19:01:21.270094154 (273h49m48.196388852s) [upload=219293]: Waiting for error group managing segments... Jul 4 19:01:21.273118218 (273h49m48.199412896s) [noise=true , address=45.91.116.8:4045 , piece_id=UWHT76QHI3LRPAMK5PJFFGEBEGYSBKZT7NIOZLNJ3ZEHNSGY3XZA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274134778 (273h49m48.200429455s) [noise=true , address=77.90.31.15:4698 , piece_id=GI4JW2FU7LJKRMQZGVXFI644BCTSIRSU5USWU6UPEWMOKTJFRMPQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.274651598 (273h49m48.200946274s) [noise=true , address=45.91.116.13:4154 , piece_id=AOI47TIUOFUJF2LTM5QD3XSGX76JQ6GN5OIQI5NSXZMEOS4AISUQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275183968 (273h49m48.201478644s) [noise=true , address=45.91.116.14:4190 , piece_id=AU3MKSFFWPQM3IGMGO5P35GDJWJ2YH4Q6Z6CGFGQEPXLAOMEIU3A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.275667848 (273h49m48.201962529s) [noise=true , address=77.90.31.14:4663 , piece_id=TSM3SKWUIMGBEZASHCKFTUR3DBHUI7ZRIEV3KI5CJIPJBQYTU4WA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.276762572 (273h49m48.203057252s) [noise=true , address=77.90.31.10:4519 , piece_id=UY4OT4LI3E7ZTFE6W6M4RYGNRQ7Z66FPJ3MY6DIUYYCR7E62MF5A, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277227148 (273h49m48.203521917s) [noise=true , address=104.36.82.8:4971 , piece_id=BTWYZKLBSRI6WHIMDPYE6TUEQJDXIAPDSX7G3KVO2VJ7RWRQN46Q, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.277685924 (273h49m48.203980601s) [noise=true , address=104.36.82.2:4729 , piece_id=FJDRP3KQDJL65Y27LE5U5SIGOWVNLYBIBJQP33F4WT5LISDG24KQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278043283 (273h49m48.204337959s) [noise=true , address=104.36.82.2:4732 , piece_id=2GOGXU22H7ZWI5OKAYFEJAQWH3XCZENYC7UJ5TED6AZ4JO7LUIBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278457355 (273h49m48.204752031s) [noise=true , address=104.36.82.11:4114 , piece_id=BQJ3TD2UXQK6S7KVBZUFU6H5GCNSOKMAD2E5SKDCIPVNPIGN4FMQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.278921322 (273h49m48.205215998s) [noise=true , address=77.90.31.14:4661 , piece_id=OP53RVAICXA6B25XCGAR73MGV5QCNLNXJCRN2OJGICS3B4MIC6VA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279552774 (273h49m48.205847454s) [noise=true , address=104.36.82.4:4803 , piece_id=DXKD2ZPSZ4XUNLDD5CDDW5ZEXJT46D76OGCIVFHNJDTWT7EB2GPA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279972556 (273h49m48.206267237s) [noise=true , address=104.36.82.13:4186 , piece_id=LGML4LYIFA3STL3CWOVSPHID4E42DUU5FM3BWOPWKE43XLYVXPIA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: <nil> Jul 4 19:01:21.279987509 (273h49m48.206282197s) [seg_pos={0 0}, upload=219293]: Segment reached optimal threshold of 13 pieces. Jul 4 19:01:21.280114273 (273h49m48.206408958s) [noise=true , address=104.36.82.10:4079 , piece_id=3OOR5S6RIUJI3P3CB2Z7Q7G7OFRZS3RJDHQFUW5Z4ASX2SFU4IOA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280222979 (273h49m48.206517655s) [noise=true , address=45.91.116.5:4298 , piece_id=REAGMJZIMQW7ENSBSYN6QVKUNIHUEUCXGALHIQCPBR6VFI5OVCBQ, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280299455 (273h49m48.206594131s) [noise=true , address=45.91.116.10:4082 , piece_id=MNOF3KZUVXZOCVOXT3MI6L6S6BFNYYQ6MTOXDA6Z37SRZTTQMFHA, seg_pos={0 0}, upload=219293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jul 4 19:01:21.280311145 (273h49m48.206605831s) [seg_pos={0 0}, upload=219293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 19:01:21.280398041 (273h49m48.206692722s) [upload=219293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 19:01:21.280404722 (273h49m48.206699412s) [upload=219293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 19:01:21.407841271 (273h49m48.334135968s) [upload=219293]: Done uploading segments... Jul 4 19:01:21.407865432 (273h49m48.334160122s) [upload=219293]: Upload finished. err: <nil> Duration: 30.000857359s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 310.975562ms FAIL (2025-07-04 17:45:10 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 311.003083ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 221.228623ms FAIL (2025-07-04 17:44:33 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 221.273502ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 204.485099ms FAIL (2025-07-04 17:43:39 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 204.515708ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 254.099628ms FAIL (2025-07-04 17:42:45 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 254.130403ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 438.182359ms FAIL (2025-07-04 17:42:26 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 438.209797ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 218.493985ms FAIL (2025-07-04 17:41:35 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 218.52158ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 426.910196ms FAIL (2025-07-04 17:40:26 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 426.942059ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 304.942724ms FAIL (2025-07-04 17:39:38 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 304.99153ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 276.576068ms FAIL (2025-07-04 17:39:03 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 276.603755ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 248.04352ms FAIL (2025-07-04 17:38:30 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 248.076656ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 253.306598ms FAIL (2025-07-04 17:37:38 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 253.332726ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 236.010186ms FAIL (2025-07-04 17:37:14 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 236.032556ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 368.227119ms FAIL (2025-07-04 17:36:03 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 368.267847ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 755.958279ms FAIL (2025-07-04 17:35:44 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 755.988259ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 219.803857ms FAIL (2025-07-04 17:34:43 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 219.8504ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 240.636071ms FAIL (2025-07-04 17:34:18 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 240.663594ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 247.813577ms FAIL (2025-07-04 17:34:00 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 247.842778ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 258.567946ms FAIL (2025-07-04 17:33:01 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 258.59713ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 469.754538ms FAIL (2025-07-04 17:31:51 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 469.845195ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 260.168399ms FAIL (2025-07-04 17:31:30 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 260.205444ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 394.673892ms FAIL (2025-07-04 17:30:47 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 394.810636ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 245.730437ms FAIL (2025-07-04 17:29:48 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 245.768817ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 270.969647ms FAIL (2025-07-04 17:29:00 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 271.002558ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 250.535552ms FAIL (2025-07-04 17:27:55 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 250.588319ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 243.142108ms FAIL (2025-07-04 17:27:11 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 243.203243ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 269.873055ms FAIL (2025-07-04 17:26:34 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 269.918291ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 466.37359ms FAIL (2025-07-04 17:26:16 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 466.402064ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 226.700553ms FAIL (2025-07-04 17:25:19 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 4 16:54:24.446974089 (271h42m51.373268785s) [upload=217578]: Done starting upload Jul 4 16:54:24.447075250 (271h42m51.373369922s) [upload=217578]: Uploading segments... Jul 4 16:54:24.447176120 (271h42m51.373470814s) [upload=217578]: Got next segment. Inline: true Jul 4 16:54:24.447181635 (271h42m51.373476312s) [upload=217578]: Waiting for error group managing segments... Jul 4 16:54:24.447223385 (271h42m51.373518062s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:24.447228416 (271h42m51.373523102s) [upload=217578]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:24.447231590 (271h42m51.373526277s) [upload=217578]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:24.547434376 (271h42m51.473729073s) [upload=217578]: Done uploading segments... Jul 4 16:54:24.547461753 (271h42m51.473756443s) [upload=217578]: Upload finished. err: <nil> Jul 4 16:54:50.012334956 (271h43m16.938629643s) [upload=217583]: Starting upload Jul 4 16:54:50.012433543 (271h43m16.938728215s) [upload=217583]: Done starting upload Jul 4 16:54:50.012522785 (271h43m16.938817456s) [upload=217583]: Uploading segments... Jul 4 16:54:50.012591678 (271h43m16.938886376s) [upload=217583]: Got next segment. Inline: true Jul 4 16:54:50.012598154 (271h43m16.938892842s) [upload=217583]: Waiting for error group managing segments... Jul 4 16:54:50.012662309 (271h43m16.938957005s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:54:50.012668193 (271h43m16.938962881s) [upload=217583]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:54:50.012671924 (271h43m16.938966643s) [upload=217583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:54:50.084069967 (271h43m17.010364644s) [upload=217583]: Done uploading segments... Jul 4 16:54:50.084093930 (271h43m17.010388611s) [upload=217583]: Upload finished. err: <nil> Jul 4 16:55:34.728175858 (271h44m1.654470546s) [upload=217594]: Starting upload Jul 4 16:55:34.728267786 (271h44m1.654562463s) [upload=217594]: Done starting upload Jul 4 16:55:34.728356170 (271h44m1.654650849s) [upload=217594]: Uploading segments... Jul 4 16:55:34.728396261 (271h44m1.654690957s) [upload=217594]: Got next segment. Inline: true Jul 4 16:55:34.728401326 (271h44m1.654696014s) [upload=217594]: Waiting for error group managing segments... Jul 4 16:55:34.728458939 (271h44m1.654753627s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:55:34.728465387 (271h44m1.654760082s) [upload=217594]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:55:34.728469617 (271h44m1.654764311s) [upload=217594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:55:34.787990513 (271h44m1.714285196s) [upload=217594]: Done uploading segments... Jul 4 16:55:34.788012542 (271h44m1.714307218s) [upload=217594]: Upload finished. err: <nil> Jul 4 16:56:22.832810212 (271h44m49.759104889s) [upload=217605]: Starting upload Jul 4 16:56:22.832893787 (271h44m49.759188475s) [upload=217605]: Done starting upload Jul 4 16:56:22.832970874 (271h44m49.759265552s) [upload=217605]: Uploading segments... Jul 4 16:56:22.833039764 (271h44m49.759334441s) [upload=217605]: Got next segment. Inline: true Jul 4 16:56:22.833044960 (271h44m49.759339636s) [upload=217605]: Waiting for error group managing segments... Jul 4 16:56:22.833137558 (271h44m49.759432234s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:56:22.833153249 (271h44m49.759447925s) [upload=217605]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:56:22.833157111 (271h44m49.759451787s) [upload=217605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:56:23.004401233 (271h44m49.930695931s) [upload=217605]: Done uploading segments... Jul 4 16:56:23.004419935 (271h44m49.930714625s) [upload=217605]: Upload finished. err: <nil> Jul 4 16:57:14.763915698 (271h45m41.690210394s) [upload=217615]: Starting upload Jul 4 16:57:14.764018476 (271h45m41.690313164s) [upload=217615]: Done starting upload Jul 4 16:57:14.764146620 (271h45m41.690441308s) [upload=217615]: Uploading segments... Jul 4 16:57:14.764203672 (271h45m41.690498372s) [upload=217615]: Got next segment. Inline: true Jul 4 16:57:14.764209947 (271h45m41.69050464s) [upload=217615]: Waiting for error group managing segments... Jul 4 16:57:14.764264588 (271h45m41.690559282s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:57:14.764272240 (271h45m41.690566938s) [upload=217615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:57:14.764276629 (271h45m41.690571324s) [upload=217615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:57:14.839328938 (271h45m41.765623632s) [upload=217615]: Done uploading segments... Jul 4 16:57:14.839353527 (271h45m41.765648218s) [upload=217615]: Upload finished. err: <nil> Jul 4 16:58:13.609116264 (271h46m40.535410958s) [upload=217628]: Starting upload Jul 4 16:58:13.609216934 (271h46m40.535511635s) [upload=217628]: Done starting upload Jul 4 16:58:13.609323370 (271h46m40.535618043s) [upload=217628]: Uploading segments... Jul 4 16:58:13.609482428 (271h46m40.535777123s) [upload=217628]: Got next segment. Inline: true Jul 4 16:58:13.609490024 (271h46m40.535784724s) [upload=217628]: Waiting for error group managing segments... Jul 4 16:58:13.609565286 (271h46m40.535859984s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:58:13.609588632 (271h46m40.535883321s) [upload=217628]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:58:13.609593329 (271h46m40.535888026s) [upload=217628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:58:13.714383856 (271h46m40.640678532s) [upload=217628]: Done uploading segments... Jul 4 16:58:13.714402167 (271h46m40.640696843s) [upload=217628]: Upload finished. err: <nil> Jul 4 16:59:15.655568095 (271h47m42.581862791s) [upload=217643]: Starting upload Jul 4 16:59:15.655648174 (271h47m42.58194286s) [upload=217643]: Done starting upload Jul 4 16:59:15.655768989 (271h47m42.582063668s) [upload=217643]: Uploading segments... Jul 4 16:59:15.655816161 (271h47m42.582110857s) [upload=217643]: Got next segment. Inline: true Jul 4 16:59:15.655821399 (271h47m42.582116086s) [upload=217643]: Waiting for error group managing segments... Jul 4 16:59:15.655882464 (271h47m42.582177151s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 16:59:15.655887954 (271h47m42.58218264s) [upload=217643]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 16:59:15.655891593 (271h47m42.582186281s) [upload=217643]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 16:59:15.714639935 (271h47m42.64093462s) [upload=217643]: Done uploading segments... Jul 4 16:59:15.714662737 (271h47m42.640957424s) [upload=217643]: Upload finished. err: <nil> Jul 4 17:00:14.409573054 (271h48m41.335867746s) [upload=217652]: Starting upload Jul 4 17:00:14.409719167 (271h48m41.336013857s) [upload=217652]: Done starting upload Jul 4 17:00:14.409888828 (271h48m41.336183517s) [upload=217652]: Uploading segments... Jul 4 17:00:14.409983947 (271h48m41.336278646s) [upload=217652]: Got next segment. Inline: true Jul 4 17:00:14.409991581 (271h48m41.336286278s) [upload=217652]: Waiting for error group managing segments... Jul 4 17:00:14.410085931 (271h48m41.336380621s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:14.410092746 (271h48m41.336387445s) [upload=217652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:14.410096964 (271h48m41.336391663s) [upload=217652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:14.532446409 (271h48m41.45874109s) [upload=217652]: Done uploading segments... Jul 4 17:00:14.532470733 (271h48m41.458765407s) [upload=217652]: Upload finished. err: <nil> Jul 4 17:00:45.857211442 (271h49m12.783506139s) [upload=217661]: Starting upload Jul 4 17:00:45.857307182 (271h49m12.783601872s) [upload=217661]: Done starting upload Jul 4 17:00:45.857382792 (271h49m12.783677464s) [upload=217661]: Uploading segments... Jul 4 17:00:45.857434180 (271h49m12.783728878s) [upload=217661]: Got next segment. Inline: true Jul 4 17:00:45.857442206 (271h49m12.783736906s) [upload=217661]: Waiting for error group managing segments... Jul 4 17:00:45.857496115 (271h49m12.783790805s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:00:45.857502240 (271h49m12.783796936s) [upload=217661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:00:45.857506169 (271h49m12.783800861s) [upload=217661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:00:45.931783557 (271h49m12.85807828s) [upload=217661]: Done uploading segments... Jul 4 17:00:45.931806872 (271h49m12.858101559s) [upload=217661]: Upload finished. err: <nil> Jul 4 17:01:53.515875549 (271h50m20.442170236s) [upload=217673]: Starting upload Jul 4 17:01:53.515960925 (271h50m20.442255601s) [upload=217673]: Done starting upload Jul 4 17:01:53.516035601 (271h50m20.442330274s) [upload=217673]: Uploading segments... Jul 4 17:01:53.516081946 (271h50m20.442376645s) [upload=217673]: Got next segment. Inline: true Jul 4 17:01:53.516088284 (271h50m20.442382976s) [upload=217673]: Waiting for error group managing segments... Jul 4 17:01:53.516142954 (271h50m20.442437651s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 17:01:53.516149778 (271h50m20.442444473s) [upload=217673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 4 17:01:53.516153718 (271h50m20.442448415s) [upload=217673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 17:01:53.599167024 (271h50m20.525461721s) [upload=217673]: Done uploading segments... Jul 4 17:01:53.599188431 (271h50m20.525483127s) [upload=217673]: Upload finished. err: <nil> Duration: 226.736204ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)