Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XX.....................X...................................] duration: 497.725514ms FAIL (2025-07-28 02:04:50 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 497.756129ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..XX.X..X.X..............................................] duration: 959.754899ms FAIL (2025-07-28 02:04:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 959.870392ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.............................................] duration: 1.23360468s FAIL (2025-07-28 02:04:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1.233639737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X..XX.X..X.X.............................................] duration: 522.542262ms FAIL (2025-07-28 02:03:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 522.574919ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.XX.....................X..................................] duration: 420.845475ms FAIL (2025-07-28 02:03:42 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 420.874824ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX............................................] duration: 3.526871611s FAIL (2025-07-28 02:03:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.526896429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.XX.....................X.................................] duration: 469.504383ms FAIL (2025-07-28 02:03:19 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 469.533777ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...........................................] duration: 6.268043995s FAIL (2025-07-28 02:03:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 6.268104239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.XX.....................X................................] duration: 378.774389ms FAIL (2025-07-28 02:02:47 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 378.802349ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..........................................] duration: 839.442766ms FAIL (2025-07-28 02:02:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 839.496883ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X..XX.X..X.X............................................] duration: 431.67965ms FAIL (2025-07-28 02:02:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.707022ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.........................................] duration: 3.266904484s FAIL (2025-07-28 02:02:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.266942268s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X..XX.X..X.X...........................................] duration: 450.182435ms FAIL (2025-07-28 02:01:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.221073ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.XX.....................X...............................] duration: 587.909738ms FAIL (2025-07-28 02:01:33 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 587.942171ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX........................................] duration: 1.303543721s FAIL (2025-07-28 02:01:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1.303574298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.XX.....................X..............................] duration: 433.604805ms FAIL (2025-07-28 02:01:18 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 433.652694ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X..XX.X..X.X..........................................] duration: 495.82256ms FAIL (2025-07-28 02:00:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 495.849648ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X....X.XX.....................X.............................] duration: 478.048078ms FAIL (2025-07-28 02:00:44 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 478.080691ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.......................................] duration: 7.640036887s FAIL (2025-07-28 02:00:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 7.640071325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X....X.XX.....................X............................] duration: 385.619832ms FAIL (2025-07-28 02:00:21 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 385.653332ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX......................................] duration: 5.750462252s FAIL (2025-07-28 02:00:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.750498301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X..XX.X..X.X.........................................] duration: 397.600877ms FAIL (2025-07-28 01:59:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 397.637765ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.X....X.XX.....................X...........................] duration: 355.883229ms FAIL (2025-07-28 01:59:24 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 355.9093ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 204.162574ms FAIL (2025-07-28 01:59:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 204.193524ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.....................................] duration: 7.976748389s FAIL (2025-07-28 01:58:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 7.976783672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X..XX.X..X.X........................................] duration: 414.693348ms FAIL (2025-07-28 01:58:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 414.73469ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 213.364994ms FAIL (2025-07-28 01:58:30 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 213.417549ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.X....X.XX.....................X..........................] duration: 396.422153ms FAIL (2025-07-28 01:58:23 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 396.454034ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X..XX.X..X.X.......................................] duration: 391.277639ms FAIL (2025-07-28 01:57:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 391.324904ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX....................................] duration: 3.60471273s FAIL (2025-07-28 01:57:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.604747189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX...................................] duration: 7.123968517s FAIL (2025-07-28 01:57:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 7.124073928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X..XX.X..X.X......................................] duration: 477.334823ms FAIL (2025-07-28 01:57:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.36896ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 192.370687ms FAIL (2025-07-28 01:57:25 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 192.395977ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.X....X.XX.....................X.........................] duration: 424.683706ms FAIL (2025-07-28 01:57:21 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 424.717243ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 246.129525ms FAIL (2025-07-28 01:57:07 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 246.160644ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X..XX.X..X.X.....................................] duration: 440.8794ms FAIL (2025-07-28 01:57:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 440.911735ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.X....X.XX.....................X........................] duration: 697.967049ms FAIL (2025-07-28 01:56:47 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 698.005213ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X..XX.X..X.X....................................] duration: 576.426121ms FAIL (2025-07-28 01:56:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 576.453374ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX..................................] duration: 3.455363373s FAIL (2025-07-28 01:56:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.455402676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 214.699847ms FAIL (2025-07-28 01:56:30 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 214.732691ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X..XX.X..X.X...................................] duration: 437.046004ms FAIL (2025-07-28 01:56:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.081894ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.................................] duration: 2.090836044s FAIL (2025-07-28 01:55:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.09087343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.X....X.XX.....................X.......................] duration: 433.377709ms FAIL (2025-07-28 01:55:46 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 433.409648ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 346.946708ms FAIL (2025-07-28 01:55:38 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 346.984174ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X..XX.X..X.X..................................] duration: 633.395599ms FAIL (2025-07-28 01:55:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 633.419348ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 348.963853ms FAIL (2025-07-28 01:54:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 349.00614ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX................................] duration: 1.638232s FAIL (2025-07-28 01:54:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1.638259535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.X....X.XX.....................X......................] duration: 685.229339ms FAIL (2025-07-28 01:54:44 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 685.270376ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X..XX.X..X.X.................................] duration: 499.029726ms FAIL (2025-07-28 01:54:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 499.122725ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 357.936537ms FAIL (2025-07-28 01:54:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 357.984319ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)