Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X....XX........X..X...........X.X....X.XX..................] duration: 601.423237ms FAIL (2025-07-28 01:34:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 601.454758ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................X..XX.X..X.X.......] duration: 460.262782ms FAIL (2025-07-28 01:34:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 460.298736ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 3.722286019s FAIL (2025-07-28 01:34:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.722320702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X............X..........................] duration: 224.216528ms FAIL (2025-07-28 01:34:00 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 224.250643ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X....XX........X..X...........X.X....X.XX.................] duration: 608.295949ms FAIL (2025-07-28 01:33:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 608.324777ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................X..XX.X..X.X......] duration: 465.003657ms FAIL (2025-07-28 01:33:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 465.037555ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 14.320215048s FAIL (2025-07-28 01:33:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 14.32024719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X....XX........X..X...........X.X....X.XX................] duration: 586.490197ms FAIL (2025-07-28 01:33:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 586.525492ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X............X.........................] duration: 201.000442ms FAIL (2025-07-28 01:32:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 201.056605ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................X..XX.X..X.X.....] duration: 417.509522ms FAIL (2025-07-28 01:32:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.546007ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X............X........................] duration: 208.71486ms FAIL (2025-07-28 01:32:22 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 208.745276ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X....XX........X..X...........X.X....X.XX...............] duration: 455.239538ms FAIL (2025-07-28 01:32:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.273988ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 14.636633179s FAIL (2025-07-28 01:31:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 14.636659089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X............X.......................] duration: 181.199586ms FAIL (2025-07-28 01:31:56 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 181.290676ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................X..XX.X..X.X....] duration: 513.525016ms FAIL (2025-07-28 01:31:49 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 513.557801ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................X............X......................] duration: 222.817615ms FAIL (2025-07-28 01:31:26 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 222.853996ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 13.275054956s FAIL (2025-07-28 01:31:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 13.275132376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X....XX........X..X...........X.X....X.XX..............] duration: 585.855454ms FAIL (2025-07-28 01:31:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 585.88793ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X............................................X..XX.X..X.X...] duration: 486.389236ms FAIL (2025-07-28 01:30:42 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 486.421676ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................X............X.....................] duration: 195.341958ms FAIL (2025-07-28 01:30:28 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 195.394622ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 4.35757916s FAIL (2025-07-28 01:30:02 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 4.357700678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X............................................X..XX.X..X.X..] duration: 574.899711ms FAIL (2025-07-28 01:30:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 574.931651ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X....XX........X..X...........X.X....X.XX.............] duration: 479.888526ms FAIL (2025-07-28 01:29:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 480.002735ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................X............X....................] duration: 184.856023ms FAIL (2025-07-28 01:29:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 184.958191ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 26.072925521s FAIL (2025-07-28 01:29:38 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 26.072947997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X............................................X..XX.X..X.X.] duration: 611.905419ms FAIL (2025-07-28 01:29:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 611.923924ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X....XX........X..X...........X.X....X.XX............] duration: 534.945057ms FAIL (2025-07-28 01:29:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 534.980817ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X............................................X..XX.X..X.X] duration: 30.039335984s FAIL (2025-07-28 01:29:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.039372187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027407208s FAIL (2025-07-28 01:29:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.027444008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................X............X...................] duration: 252.021784ms FAIL (2025-07-28 01:28:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 252.079446ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [........X....XX........X..X...........X.X....X.XX...........] duration: 493.74067ms FAIL (2025-07-28 01:28:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 493.772339ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................X............X..................] duration: 351.498528ms FAIL (2025-07-28 01:28:33 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 351.514715ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................X............X.................] duration: 416.271973ms FAIL (2025-07-28 01:28:18 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 416.382101ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027689793s FAIL (2025-07-28 01:28:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== VKT375DMNA, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.780631007 (27h35m28.561564385s) [noise=true , address=104.36.82.54:4183 , piece_id=R5XAFROTSIQ5D4URZISM2Q6DMWPJQM65BOQMR2YITPZXM5EGDC7Q, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.780919936 (27h35m28.56185332s) [noise=true , address=104.36.82.15:4271 , piece_id=RES3BBF3UAJ3URK36RS6O5LIMK6CKFYBSK3HK3S6I7PAHZNVEC4Q, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.781078065 (27h35m28.56201145s) [noise=true , address=104.36.82.10:4089 , piece_id=PQEQEJXC7LYU7EZYRGE5FAIW7R44FAHTP4TT5DGIOBXCEH6VLWVA, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.781350579 (27h35m28.562283966s) [noise=true , address=79.127.201.215:4355 , piece_id=UNGDMBXGFCJMFNSF5XJJHU3TR7QQWBLW2L2EF4OLDKPXLL22AHVA, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.781505374 (27h35m28.562438754s) [noise=true , address=104.36.82.12:4180 , piece_id=QD2IVRTSPXT3I5WFRSIVLMKSEFUDEJSVU5IHBDHI2SCPSTCRMMEA, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.781788699 (27h35m28.562722084s) [noise=true , address=104.36.82.52:4103 , piece_id=DZWDQTAHILIR2VORANX65TRH6MSYRAJZPUI425C5FIBPDUTHKSMA, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.781939912 (27h35m28.562873297s) [noise=true , address=104.36.82.19:4403 , piece_id=BBGCUND2RTP7RMZIMUAF4DCULI6HFIC3AXJRPXZUSBM563QKHQIA, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.782218210 (27h35m28.563151592s) [noise=true , address=64.49.13.140:4508 , piece_id=24LBYKXUENOM2BKTLYIHFBYCDLHRRQ6M7L2MN5XJKZ6AM4RVTLOQ, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.782530984 (27h35m28.563464368s) [noise=true , address=79.127.205.245:4246 , piece_id=LIZQMA4UAAVGP536EGWOWYL2YLPCX5B75RJRX5DG6BHCDFG6DURQ, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.782643163 (27h35m28.563576546s) [noise=true , address=104.36.82.51:4052 , piece_id=RB6P2PQFIK2NVGNQAH7O5C7UOGPOTERWISY6SZ5DJKAYU3MPDE7A, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.782966419 (27h35m28.563899798s) [noise=true , address=64.49.13.138:4482 , piece_id=UFYDQRR6HZQCHFAC256LBGWQRWN7EWJIXPVNVTHYYDAQWGUDGNSA, seg_pos={0 0}, upload=26317]: Uploading piece... Jul 28 01:27:12.992587403 (27h35m28.773520767s) [upload=26317]: Next returned nil segment Jul 28 01:27:12.992606516 (27h35m28.773539892s) [upload=26317]: Waiting for error group managing segments... Jul 28 01:27:13.388039280 (27h35m29.168972644s) [noise=true , address=79.127.205.245:4246 , piece_id=LIZQMA4UAAVGP536EGWOWYL2YLPCX5B75RJRX5DG6BHCDFG6DURQ, seg_pos={0 0}, upload=26317]: Done uploading piece. err: <nil> Jul 28 01:27:13.766575512 (27h35m29.547508889s) [noise=true , address=64.49.13.140:4508 , piece_id=24LBYKXUENOM2BKTLYIHFBYCDLHRRQ6M7L2MN5XJKZ6AM4RVTLOQ, seg_pos={0 0}, upload=26317]: Done uploading piece. err: <nil> Jul 28 01:27:14.676921542 (27h35m30.457854912s) [noise=true , address=79.127.201.215:4355 , piece_id=UNGDMBXGFCJMFNSF5XJJHU3TR7QQWBLW2L2EF4OLDKPXLL22AHVA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: <nil> Jul 28 01:27:18.713784531 (27h35m34.494717898s) [noise=true , address=64.49.13.138:4482 , piece_id=UFYDQRR6HZQCHFAC256LBGWQRWN7EWJIXPVNVTHYYDAQWGUDGNSA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: <nil> Jul 28 01:28:12.704279270 (27h36m28.485212637s) [noise=true , address=104.36.82.13:4215 , piece_id=AU24TU5RDFHCTRGUXPYHKHNWTJ7RWEMQOKO23AIGN4NTFJCH5XKA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:1vV1hGhZqeEoh5ei7yRu49oyAvTZ2T7S1x3ooufJPtFH2TGBkQ, address:104.36.82.13:4215): protocol: rpc: dial tcp 104.36.82.13:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.704390416 (27h36m28.485323788s) [noise=true , address=104.36.82.54:4183 , piece_id=R5XAFROTSIQ5D4URZISM2Q6DMWPJQM65BOQMR2YITPZXM5EGDC7Q, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: rpc: dial tcp 104.36.82.54:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.704463359 (27h36m28.485396742s) [noise=true , address=104.36.82.2:4754 , piece_id=MDITMXB4BFEEDK4OCJJ23YIWALJ4DQIY5HCWNFZ7LEFOHKRVB3OA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:125aPwoHeFabk6zZ6bkGNj3USJYgTQG3RB1QhhXssj7Vik2AuCs, address:104.36.82.2:4754): protocol: rpc: dial tcp 104.36.82.2:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.704521080 (27h36m28.485454444s) [noise=true , address=104.36.82.14:4230 , piece_id=D7RI5EDHPPBSULTNLA5MWAZT7N3WJNLO54PJRMHRZYH32JM5DVLA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:15hXnJMgf7FAJJh5y1S35eNLdvvqQR4yFot9M8FAmwJDyYVsWf, address:104.36.82.14:4230): protocol: rpc: dial tcp 104.36.82.14:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.704575272 (27h36m28.485508638s) [noise=true , address=104.36.82.53:4134 , piece_id=W5X6X76W7UR56IQ3BZ54GY2XWI33MPZ4KNKWHYJQYZVKT375DMNA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: rpc: dial tcp 104.36.82.53:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.704846899 (27h36m28.485780262s) [noise=true , address=104.36.82.17:4358 , piece_id=JHZRJV42DUI5HI2HTQ7UIX3FW7PQCWBIOWLA4IHG36YEV4AWH7LA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.704971466 (27h36m28.485904829s) [noise=true , address=104.36.82.45:4906 , piece_id=WDISREUZMZYJUY5C2PZ6KHPQMK7L46KYQGCPNJOBCAX4DMXLW2SQ, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:104.36.82.45:4906): protocol: rpc: dial tcp 104.36.82.45:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.705077983 (27h36m28.486011357s) [noise=true , address=104.36.82.15:4271 , piece_id=RES3BBF3UAJ3URK36RS6O5LIMK6CKFYBSK3HK3S6I7PAHZNVEC4Q, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu, address:104.36.82.15:4271): protocol: rpc: dial tcp 104.36.82.15:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.705200237 (27h36m28.486133602s) [noise=true , address=104.36.82.19:4403 , piece_id=BBGCUND2RTP7RMZIMUAF4DCULI6HFIC3AXJRPXZUSBM563QKHQIA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: rpc: dial tcp 104.36.82.19:4403: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.705393211 (27h36m28.486326588s) [noise=true , address=104.36.82.10:4089 , piece_id=PQEQEJXC7LYU7EZYRGE5FAIW7R44FAHTP4TT5DGIOBXCEH6VLWVA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: rpc: dial tcp 104.36.82.10:4089: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.705471161 (27h36m28.486404545s) [noise=true , address=104.36.82.12:4180 , piece_id=QD2IVRTSPXT3I5WFRSIVLMKSEFUDEJSVU5IHBDHI2SCPSTCRMMEA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:4180): protocol: rpc: dial tcp 104.36.82.12:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.705529870 (27h36m28.486463232s) [noise=true , address=104.36.82.22:4541 , piece_id=VK2PH5HEXAGFXU4KD3KKTLAEPXBNNPJPPETT4GT2ARFMUJC6EKHQ, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:1zupYUhZUbuwMv5gwprk8M3kPYQ6qJiFK8ZsgGg3b1sqH6UwGr, address:104.36.82.22:4541): protocol: rpc: dial tcp 104.36.82.22:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.705564515 (27h36m28.486497881s) [noise=true , address=104.36.82.52:4103 , piece_id=DZWDQTAHILIR2VORANX65TRH6MSYRAJZPUI425C5FIBPDUTHKSMA, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: rpc: dial tcp 104.36.82.52:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.705624045 (27h36m28.486557414s) [noise=true , address=104.36.82.51:4052 , piece_id=RB6P2PQFIK2NVGNQAH7O5C7UOGPOTERWISY6SZ5DJKAYU3MPDE7A, seg_pos={0 0}, upload=26317]: Done uploading piece. err: ecclient: upload failed (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V, address:104.36.82.51:4052): protocol: write tcp 159.223.143.39:33604->104.36.82.51:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:28:12.705659348 (27h36m28.486592727s) [seg_pos={0 0}, upload=26317]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 28 01:28:12.729832313 (27h36m28.510765678s) [upload=26317]: Done uploading segments... Jul 28 01:28:12.729865174 (27h36m28.51079855s) [upload=26317]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.027709037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...X............................................X..XX.X..X.] duration: 608.326525ms FAIL (2025-07-28 01:27:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 5GQSVTWSTEBJ5UP35RYYMCQYXJQ, seg_pos={0 0}, upload=26305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 159.223.143.39:59438->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:37.420788083 (27h34m53.201721447s) [seg_pos={0 0}, upload=26305]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:26:37.420854573 (27h34m53.201787949s) [upload=26305]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:26:37.420859373 (27h34m53.201792749s) [upload=26305]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:26:37.466616399 (27h34m53.247549783s) [upload=26305]: Done uploading segments... Jul 28 01:26:37.466642018 (27h34m53.247575399s) [upload=26305]: Upload finished. err: <nil> Jul 28 01:26:55.267787669 (27h35m11.048721054s) [upload=26311]: Starting upload Jul 28 01:26:55.267980042 (27h35m11.048913427s) [upload=26311]: Done starting upload Jul 28 01:26:55.268322890 (27h35m11.049256268s) [upload=26311]: Uploading segments... Jul 28 01:26:55.269601674 (27h35m11.050535044s) [upload=26311]: Got next segment. Inline: false Jul 28 01:26:55.269727892 (27h35m11.050661284s) [upload=26311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:26:55.269736407 (27h35m11.050669789s) [upload=26311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:26:55.315451694 (27h35m11.096385059s) [seg_pos={0 0}, upload=26311]: Begin upload segment... Jul 28 01:26:55.315492374 (27h35m11.09642575s) [seg_pos={0 0}, upload=26311]: Done begin upload segment. Jul 28 01:26:55.315514063 (27h35m11.096447441s) [noise=true , address=77.90.31.13:4626 , piece_id=CKQIKKMGQOFGZYYJBGPZAUCOL4ORWXGTE3NP4N4US5W5H5Q6Z6RA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.317610761 (27h35m11.098544144s) [noise=true , address=45.91.116.12:4227 , piece_id=J2UJFSSBPWAVAQE5VV5KTM5IVMBKPF5VZURWDJX7XPFQ7C3UIB4A, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.318164879 (27h35m11.099098262s) [noise=true , address=45.91.116.11:4119 , piece_id=F6F5KNB5BOE2U7TQQPRLYRJPVSNTK34NBKOKXDJO3TWVCAK46ZOA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.318500169 (27h35m11.099433546s) [noise=true , address=77.90.31.11:4553 , piece_id=UE2C5YTRJAYCQTTWZ5REYDZMUZ55FUNWE5K3PLX3WCZZ5UXMI5KA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.318784358 (27h35m11.099717723s) [noise=true , address=77.90.31.14:4661 , piece_id=JZYPFZUSP3YRM3675HHDJM66LUMT27UOKBQLA6MQ63LSVSD2NGLA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.319089274 (27h35m11.10002265s) [noise=true , address=104.36.82.7:4911 , piece_id=KGBGZP5TTTXI5V5YPMKNOK26HRSJSTO3WM2PGTUCMBNOGBHTSLBQ, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.319427644 (27h35m11.100361022s) [noise=true , address=77.90.31.12:4592 , piece_id=FYB3BWNTRQL4RIC53JEDTHTJIJMC3XTORCQUG2WNNZEKBKSHVTYA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.319722570 (27h35m11.100655948s) [noise=true , address=104.36.82.9:4006 , piece_id=E5CUAHSILPTH2K57XTB6JBJI3RT4KIO2M5OEH64TAVZWSUJTKBHA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.320021241 (27h35m11.100954617s) [noise=true , address=77.90.31.19:4841 , piece_id=X4ZLRATT7NTHSWYEHFJ7N6L5FCI7OQFEYFJOD6XOQSXJ4CNTIMHA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.320378348 (27h35m11.101311732s) [noise=true , address=45.91.116.6:4975 , piece_id=M2L24TEQDTCI2FXGJDF2HP7JRMH5M3WHMCZOIOYMHKT4W2RK7AHA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.322182956 (27h35m11.103116338s) [noise=true , address=45.91.116.4:4937 , piece_id=523YBOX7JE3H4N4AI5BMTWC2G3GZSLLYLQ2TD44APVO6ECRHWC7A, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.322533178 (27h35m11.103466554s) [noise=true , address=45.91.116.8:4047 , piece_id=YAQVDW6TTUS22LLJ75DC4I6U6JKQ6L5K35BP7UP2VOZAPOOB2MBA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.322874259 (27h35m11.103807644s) [noise=true , address=77.90.31.14:4663 , piece_id=PHMUA7H6WFW3VQ52ZRWO5FNJWS5NSXS7KUJ5MMYSF3KCMB2YVZFQ, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.323258303 (27h35m11.104191683s) [noise=true , address=104.36.82.15:4257 , piece_id=7E7ECYXCHARRZRGGV7PXQOTXBVYC2GHTBGFQ67XXR2ORZ5QBHLXA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.323564536 (27h35m11.104497926s) [noise=true , address=104.36.82.4:4802 , piece_id=62A6QASLN4JAQCA7FJL2FQY2AZ4ZU5YHDGLEERCEYJDKYBRKZBLQ, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.323969265 (27h35m11.104902645s) [noise=true , address=45.91.116.6:4974 , piece_id=BFUCQEOA6M4J4TQJPFTOLSRVMUPNCZZVIXK7S2AYDSCSVPIQDI2Q, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.464128908 (27h35m11.245062288s) [upload=26311]: Next returned nil segment Jul 28 01:26:55.464146990 (27h35m11.245080352s) [upload=26311]: Waiting for error group managing segments... Jul 28 01:26:55.510419125 (27h35m11.291352506s) [noise=true , address=77.90.31.13:4626 , piece_id=CKQIKKMGQOFGZYYJBGPZAUCOL4ORWXGTE3NP4N4US5W5H5Q6Z6RA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.511046055 (27h35m11.291979433s) [noise=true , address=45.91.116.12:4227 , piece_id=J2UJFSSBPWAVAQE5VV5KTM5IVMBKPF5VZURWDJX7XPFQ7C3UIB4A, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.511593608 (27h35m11.292526971s) [noise=true , address=45.91.116.11:4119 , piece_id=F6F5KNB5BOE2U7TQQPRLYRJPVSNTK34NBKOKXDJO3TWVCAK46ZOA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.514537640 (27h35m11.295471018s) [noise=true , address=77.90.31.11:4553 , piece_id=UE2C5YTRJAYCQTTWZ5REYDZMUZ55FUNWE5K3PLX3WCZZ5UXMI5KA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.517053902 (27h35m11.297987283s) [noise=true , address=77.90.31.12:4592 , piece_id=FYB3BWNTRQL4RIC53JEDTHTJIJMC3XTORCQUG2WNNZEKBKSHVTYA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.553187443 (27h35m11.33412087s) [noise=true , address=77.90.31.14:4663 , piece_id=PHMUA7H6WFW3VQ52ZRWO5FNJWS5NSXS7KUJ5MMYSF3KCMB2YVZFQ, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.553846805 (27h35m11.334780183s) [noise=true , address=45.91.116.6:4975 , piece_id=M2L24TEQDTCI2FXGJDF2HP7JRMH5M3WHMCZOIOYMHKT4W2RK7AHA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.554444872 (27h35m11.335378251s) [noise=true , address=45.91.116.6:4974 , piece_id=BFUCQEOA6M4J4TQJPFTOLSRVMUPNCZZVIXK7S2AYDSCSVPIQDI2Q, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.554783267 (27h35m11.335716636s) [noise=true , address=77.90.31.19:4841 , piece_id=X4ZLRATT7NTHSWYEHFJ7N6L5FCI7OQFEYFJOD6XOQSXJ4CNTIMHA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.555389962 (27h35m11.336323326s) [noise=true , address=45.91.116.4:4937 , piece_id=523YBOX7JE3H4N4AI5BMTWC2G3GZSLLYLQ2TD44APVO6ECRHWC7A, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.557017086 (27h35m11.337950448s) [noise=true , address=45.91.116.8:4047 , piece_id=YAQVDW6TTUS22LLJ75DC4I6U6JKQ6L5K35BP7UP2VOZAPOOB2MBA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.560530376 (27h35m11.341463754s) [noise=true , address=77.90.31.14:4661 , piece_id=JZYPFZUSP3YRM3675HHDJM66LUMT27UOKBQLA6MQ63LSVSD2NGLA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:27:25.237579360 (27h35m41.018512746s) [noise=true , address=104.36.82.7:4911 , piece_id=KGBGZP5TTTXI5V5YPMKNOK26HRSJSTO3WM2PGTUCMBNOGBHTSLBQ, seg_pos={0 0}, upload=26311]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:25.237738406 (27h35m41.018671784s) [noise=true , address=104.36.82.9:4006 , piece_id=E5CUAHSILPTH2K57XTB6JBJI3RT4KIO2M5OEH64TAVZWSUJTKBHA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:25.237754049 (27h35m41.018687467s) [noise=true , address=104.36.82.4:4802 , piece_id=62A6QASLN4JAQCA7FJL2FQY2AZ4ZU5YHDGLEERCEYJDKYBRKZBLQ, seg_pos={0 0}, upload=26311]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:25.237820076 (27h35m41.018753451s) [noise=true , address=104.36.82.15:4257 , piece_id=7E7ECYXCHARRZRGGV7PXQOTXBVYC2GHTBGFQ67XXR2ORZ5QBHLXA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:44828->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:25.237858384 (27h35m41.01879176s) [seg_pos={0 0}, upload=26311]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:27:25.263759883 (27h35m41.044693251s) [upload=26311]: Done uploading segments... Jul 28 01:27:25.263803793 (27h35m41.044737172s) [upload=26311]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 608.353846ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................X............X................] duration: 219.750105ms FAIL (2025-07-28 01:27:52 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 219.802511ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X....XX........X..X...........X.X....X.XX..........] duration: 378.935971ms FAIL (2025-07-28 01:27:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 378.970313ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...X............................................X..XX.X..X] duration: 30.028673016s FAIL (2025-07-28 01:27:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 5GQSVTWSTEBJ5UP35RYYMCQYXJQ, seg_pos={0 0}, upload=26305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 159.223.143.39:59438->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:26:37.420788083 (27h34m53.201721447s) [seg_pos={0 0}, upload=26305]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:26:37.420854573 (27h34m53.201787949s) [upload=26305]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:26:37.420859373 (27h34m53.201792749s) [upload=26305]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:26:37.466616399 (27h34m53.247549783s) [upload=26305]: Done uploading segments... Jul 28 01:26:37.466642018 (27h34m53.247575399s) [upload=26305]: Upload finished. err: <nil> Jul 28 01:26:55.267787669 (27h35m11.048721054s) [upload=26311]: Starting upload Jul 28 01:26:55.267980042 (27h35m11.048913427s) [upload=26311]: Done starting upload Jul 28 01:26:55.268322890 (27h35m11.049256268s) [upload=26311]: Uploading segments... Jul 28 01:26:55.269601674 (27h35m11.050535044s) [upload=26311]: Got next segment. Inline: false Jul 28 01:26:55.269727892 (27h35m11.050661284s) [upload=26311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:26:55.269736407 (27h35m11.050669789s) [upload=26311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:26:55.315451694 (27h35m11.096385059s) [seg_pos={0 0}, upload=26311]: Begin upload segment... Jul 28 01:26:55.315492374 (27h35m11.09642575s) [seg_pos={0 0}, upload=26311]: Done begin upload segment. Jul 28 01:26:55.315514063 (27h35m11.096447441s) [noise=true , address=77.90.31.13:4626 , piece_id=CKQIKKMGQOFGZYYJBGPZAUCOL4ORWXGTE3NP4N4US5W5H5Q6Z6RA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.317610761 (27h35m11.098544144s) [noise=true , address=45.91.116.12:4227 , piece_id=J2UJFSSBPWAVAQE5VV5KTM5IVMBKPF5VZURWDJX7XPFQ7C3UIB4A, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.318164879 (27h35m11.099098262s) [noise=true , address=45.91.116.11:4119 , piece_id=F6F5KNB5BOE2U7TQQPRLYRJPVSNTK34NBKOKXDJO3TWVCAK46ZOA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.318500169 (27h35m11.099433546s) [noise=true , address=77.90.31.11:4553 , piece_id=UE2C5YTRJAYCQTTWZ5REYDZMUZ55FUNWE5K3PLX3WCZZ5UXMI5KA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.318784358 (27h35m11.099717723s) [noise=true , address=77.90.31.14:4661 , piece_id=JZYPFZUSP3YRM3675HHDJM66LUMT27UOKBQLA6MQ63LSVSD2NGLA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.319089274 (27h35m11.10002265s) [noise=true , address=104.36.82.7:4911 , piece_id=KGBGZP5TTTXI5V5YPMKNOK26HRSJSTO3WM2PGTUCMBNOGBHTSLBQ, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.319427644 (27h35m11.100361022s) [noise=true , address=77.90.31.12:4592 , piece_id=FYB3BWNTRQL4RIC53JEDTHTJIJMC3XTORCQUG2WNNZEKBKSHVTYA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.319722570 (27h35m11.100655948s) [noise=true , address=104.36.82.9:4006 , piece_id=E5CUAHSILPTH2K57XTB6JBJI3RT4KIO2M5OEH64TAVZWSUJTKBHA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.320021241 (27h35m11.100954617s) [noise=true , address=77.90.31.19:4841 , piece_id=X4ZLRATT7NTHSWYEHFJ7N6L5FCI7OQFEYFJOD6XOQSXJ4CNTIMHA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.320378348 (27h35m11.101311732s) [noise=true , address=45.91.116.6:4975 , piece_id=M2L24TEQDTCI2FXGJDF2HP7JRMH5M3WHMCZOIOYMHKT4W2RK7AHA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.322182956 (27h35m11.103116338s) [noise=true , address=45.91.116.4:4937 , piece_id=523YBOX7JE3H4N4AI5BMTWC2G3GZSLLYLQ2TD44APVO6ECRHWC7A, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.322533178 (27h35m11.103466554s) [noise=true , address=45.91.116.8:4047 , piece_id=YAQVDW6TTUS22LLJ75DC4I6U6JKQ6L5K35BP7UP2VOZAPOOB2MBA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.322874259 (27h35m11.103807644s) [noise=true , address=77.90.31.14:4663 , piece_id=PHMUA7H6WFW3VQ52ZRWO5FNJWS5NSXS7KUJ5MMYSF3KCMB2YVZFQ, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.323258303 (27h35m11.104191683s) [noise=true , address=104.36.82.15:4257 , piece_id=7E7ECYXCHARRZRGGV7PXQOTXBVYC2GHTBGFQ67XXR2ORZ5QBHLXA, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.323564536 (27h35m11.104497926s) [noise=true , address=104.36.82.4:4802 , piece_id=62A6QASLN4JAQCA7FJL2FQY2AZ4ZU5YHDGLEERCEYJDKYBRKZBLQ, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.323969265 (27h35m11.104902645s) [noise=true , address=45.91.116.6:4974 , piece_id=BFUCQEOA6M4J4TQJPFTOLSRVMUPNCZZVIXK7S2AYDSCSVPIQDI2Q, seg_pos={0 0}, upload=26311]: Uploading piece... Jul 28 01:26:55.464128908 (27h35m11.245062288s) [upload=26311]: Next returned nil segment Jul 28 01:26:55.464146990 (27h35m11.245080352s) [upload=26311]: Waiting for error group managing segments... Jul 28 01:26:55.510419125 (27h35m11.291352506s) [noise=true , address=77.90.31.13:4626 , piece_id=CKQIKKMGQOFGZYYJBGPZAUCOL4ORWXGTE3NP4N4US5W5H5Q6Z6RA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.511046055 (27h35m11.291979433s) [noise=true , address=45.91.116.12:4227 , piece_id=J2UJFSSBPWAVAQE5VV5KTM5IVMBKPF5VZURWDJX7XPFQ7C3UIB4A, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.511593608 (27h35m11.292526971s) [noise=true , address=45.91.116.11:4119 , piece_id=F6F5KNB5BOE2U7TQQPRLYRJPVSNTK34NBKOKXDJO3TWVCAK46ZOA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.514537640 (27h35m11.295471018s) [noise=true , address=77.90.31.11:4553 , piece_id=UE2C5YTRJAYCQTTWZ5REYDZMUZ55FUNWE5K3PLX3WCZZ5UXMI5KA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.517053902 (27h35m11.297987283s) [noise=true , address=77.90.31.12:4592 , piece_id=FYB3BWNTRQL4RIC53JEDTHTJIJMC3XTORCQUG2WNNZEKBKSHVTYA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.553187443 (27h35m11.33412087s) [noise=true , address=77.90.31.14:4663 , piece_id=PHMUA7H6WFW3VQ52ZRWO5FNJWS5NSXS7KUJ5MMYSF3KCMB2YVZFQ, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.553846805 (27h35m11.334780183s) [noise=true , address=45.91.116.6:4975 , piece_id=M2L24TEQDTCI2FXGJDF2HP7JRMH5M3WHMCZOIOYMHKT4W2RK7AHA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.554444872 (27h35m11.335378251s) [noise=true , address=45.91.116.6:4974 , piece_id=BFUCQEOA6M4J4TQJPFTOLSRVMUPNCZZVIXK7S2AYDSCSVPIQDI2Q, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.554783267 (27h35m11.335716636s) [noise=true , address=77.90.31.19:4841 , piece_id=X4ZLRATT7NTHSWYEHFJ7N6L5FCI7OQFEYFJOD6XOQSXJ4CNTIMHA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.555389962 (27h35m11.336323326s) [noise=true , address=45.91.116.4:4937 , piece_id=523YBOX7JE3H4N4AI5BMTWC2G3GZSLLYLQ2TD44APVO6ECRHWC7A, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.557017086 (27h35m11.337950448s) [noise=true , address=45.91.116.8:4047 , piece_id=YAQVDW6TTUS22LLJ75DC4I6U6JKQ6L5K35BP7UP2VOZAPOOB2MBA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:26:55.560530376 (27h35m11.341463754s) [noise=true , address=77.90.31.14:4661 , piece_id=JZYPFZUSP3YRM3675HHDJM66LUMT27UOKBQLA6MQ63LSVSD2NGLA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: <nil> Jul 28 01:27:25.237579360 (27h35m41.018512746s) [noise=true , address=104.36.82.7:4911 , piece_id=KGBGZP5TTTXI5V5YPMKNOK26HRSJSTO3WM2PGTUCMBNOGBHTSLBQ, seg_pos={0 0}, upload=26311]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:25.237738406 (27h35m41.018671784s) [noise=true , address=104.36.82.9:4006 , piece_id=E5CUAHSILPTH2K57XTB6JBJI3RT4KIO2M5OEH64TAVZWSUJTKBHA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:25.237754049 (27h35m41.018687467s) [noise=true , address=104.36.82.4:4802 , piece_id=62A6QASLN4JAQCA7FJL2FQY2AZ4ZU5YHDGLEERCEYJDKYBRKZBLQ, seg_pos={0 0}, upload=26311]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:25.237820076 (27h35m41.018753451s) [noise=true , address=104.36.82.15:4257 , piece_id=7E7ECYXCHARRZRGGV7PXQOTXBVYC2GHTBGFQ67XXR2ORZ5QBHLXA, seg_pos={0 0}, upload=26311]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:44828->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:25.237858384 (27h35m41.01879176s) [seg_pos={0 0}, upload=26311]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:27:25.263759883 (27h35m41.044693251s) [upload=26311]: Done uploading segments... Jul 28 01:27:25.263803793 (27h35m41.044737172s) [upload=26311]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.02872365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.035414398s FAIL (2025-07-28 01:27:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 4883 , piece_id=PY652HMWVOURBVH6UPGQN2GGQD7XY6I333Q4AMHQKTBVM3B6X2FA, seg_pos={0 0}, upload=26299]: Uploading piece... Jul 28 01:26:12.694494591 (27h34m28.475427991s) [noise=true , address=104.36.82.20:4466 , piece_id=Y7MCXHPYGHPYEZBDSH2P3B6R6T3APSJRU2WWSA37FZ3NS66NELEA, seg_pos={0 0}, upload=26299]: Uploading piece... Jul 28 01:26:12.694948010 (27h34m28.475881393s) [noise=true , address=104.36.82.14:4255 , piece_id=FVMJZXGS7MPIYTUZE5FLMQ66FW54EUSUOD475CVZFGJXAGRIF65Q, seg_pos={0 0}, upload=26299]: Uploading piece... Jul 28 01:26:12.695381513 (27h34m28.47631489s) [noise=true , address=104.36.82.56:4247 , piece_id=CE3BRQ5HJ4R2YFOMHS5P4UEKWZI2VRMYEEDMX7YLJD62FTMYZ3WA, seg_pos={0 0}, upload=26299]: Uploading piece... Jul 28 01:26:12.695793626 (27h34m28.476727009s) [noise=true , address=104.36.82.52:4115 , piece_id=VPY4ZS2E3E3Z7X5B2TEAXBM3XPDBMTNFJ7DBZEN52BC2FQ3JIA6A, seg_pos={0 0}, upload=26299]: Uploading piece... Jul 28 01:26:12.696278733 (27h34m28.477212113s) [noise=true , address=104.36.82.40:4952 , piece_id=PUY4VJVEOB6OFCQ742NEKH2HHRFPH43GNZ3TOJU5YEI4VR4JY56Q, seg_pos={0 0}, upload=26299]: Uploading piece... Jul 28 01:26:12.696475160 (27h34m28.477408546s) [noise=true , address=104.36.82.3:4775 , piece_id=DFFMDWNETWJK6CCXNRJC5PB266F2ERJ2I7UWY6GQMRGXA7OUMQMQ, seg_pos={0 0}, upload=26299]: Uploading piece... Jul 28 01:26:12.697095474 (27h34m28.478028858s) [noise=true , address=104.36.82.13:4205 , piece_id=A73Q7THX3MFSKVSCMZXAYGGLKHWMKA4BMOVCWM2PTXFLLBLGE2GA, seg_pos={0 0}, upload=26299]: Uploading piece... Jul 28 01:26:12.697656586 (27h34m28.478589965s) [noise=true , address=77.90.31.15:4721 , piece_id=CF6KW25B2FNRDYOHKEGS56PJULYDFU2HMLLYXAGS7AVX44QCC7OA, seg_pos={0 0}, upload=26299]: Uploading piece... Jul 28 01:26:12.784639356 (27h34m28.565572736s) [upload=26299]: Next returned nil segment Jul 28 01:26:12.784662722 (27h34m28.565596109s) [upload=26299]: Waiting for error group managing segments... Jul 28 01:26:12.934236967 (27h34m28.715170336s) [noise=true , address=77.90.31.15:4721 , piece_id=CF6KW25B2FNRDYOHKEGS56PJULYDFU2HMLLYXAGS7AVX44QCC7OA, seg_pos={0 0}, upload=26299]: Done uploading piece. err: <nil> Jul 28 01:26:13.105707065 (27h34m28.886640431s) [noise=true , address=79.127.201.215:4352 , piece_id=EZWIE7PZLMP6MGL4ZNJKJ35VCDRGPNJWN6IVP3FEHJ34IX2ZE5YA, seg_pos={0 0}, upload=26299]: Done uploading piece. err: <nil> Jul 28 01:27:12.613630422 (27h35m28.394563807s) [noise=true , address=104.36.82.54:4188 , piece_id=7T5ZPCGZTIVWYITTQCURYPFSPZJG6G4MNCGTTCAS3OO5OPFMHFTA, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG, address:104.36.82.54:4188): protocol: rpc: dial tcp 104.36.82.54:4188: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.613728724 (27h35m28.39466209s) [noise=true , address=104.36.82.44:4883 , piece_id=PY652HMWVOURBVH6UPGQN2GGQD7XY6I333Q4AMHQKTBVM3B6X2FA, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN, address:104.36.82.44:4883): protocol: rpc: dial tcp 104.36.82.44:4883: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614086467 (27h35m28.395019844s) [noise=true , address=104.36.82.56:4247 , piece_id=CE3BRQ5HJ4R2YFOMHS5P4UEKWZI2VRMYEEDMX7YLJD62FTMYZ3WA, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:12PzKreuPT9X594ZTTYbXanNdaFVVJsnKL99fumjXymnRa9jeEq, address:104.36.82.56:4247): protocol: write tcp 159.223.143.39:33646->104.36.82.56:4247: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614152594 (27h35m28.395085957s) [noise=true , address=104.36.82.10:4094 , piece_id=P2BXUDNXJLKSULQQA6VC22JHGPY5BEFF7MSFOHNUZ3TRLAAE777Q, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:124zBCfZBTW69BSaGRzUyjWUBjSJhaLMZXQYnqMPGSKL4EX6HVV, address:104.36.82.10:4094): protocol: rpc: dial tcp 104.36.82.10:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614244485 (27h35m28.395177855s) [noise=true , address=104.36.82.40:4952 , piece_id=PUY4VJVEOB6OFCQ742NEKH2HHRFPH43GNZ3TOJU5YEI4VR4JY56Q, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:1QucqyXk8Zx4q4uK6nMULHHRjZCRfQDoUREoHZNEjsQwVQt7gK, address:104.36.82.40:4952): protocol: rpc: dial tcp 104.36.82.40:4952: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614273494 (27h35m28.395206881s) [noise=true , address=104.36.82.55:4223 , piece_id=IFOFL26VQAUUGVVHVE7EXSFG27Z3ZIIRGPUFIOGGIL42EDWXAXBQ, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: rpc: dial tcp 104.36.82.55:4223: i/o timeout; conn closed: op Jul 28 01:27:12.614317169 (27h35m28.395250534s) [noise=true , address=104.36.82.3:4775 , piece_id=DFFMDWNETWJK6CCXNRJC5PB266F2ERJ2I7UWY6GQMRGXA7OUMQMQ, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:12BvQLLnX5A2BgKeqn1ubs5iBeCUsUGH8hFjpB2HvY53YH9Nw1n, address:104.36.82.3:4775): protocol: rpc: dial tcp 104.36.82.3:4775: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614376713 (27h35m28.39531009s) [noise=true , address=104.36.82.13:4205 , piece_id=A73Q7THX3MFSKVSCMZXAYGGLKHWMKA4BMOVCWM2PTXFLLBLGE2GA, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr, address:104.36.82.13:4205): protocol: write tcp 159.223.143.39:51196->104.36.82.13:4205: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614439288 (27h35m28.395372654s) [noise=true , address=104.36.82.46:4485 , piece_id=EIPIKTI3JGFU5J7MQ5QITMHXKW76F7NN4DQK5TNBJI6OPLIJFBWA, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ, address:104.36.82.46:4485): protocol: rpc: dial tcp 104.36.82.46:4485: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614499065 (27h35m28.395432431s) [noise=true , address=104.36.82.51:4050 , piece_id=TQ4TNJA3AZDQCF7V52WSXZDIXI3KYYHGR7E3TZKKZFWR7YBI5MWQ, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:104.36.82.51:4050): protocol: rpc: dial tcp 104.36.82.51:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614532945 (27h35m28.395466316s) [noise=true , address=104.36.82.20:4466 , piece_id=Y7MCXHPYGHPYEZBDSH2P3B6R6T3APSJRU2WWSA37FZ3NS66NELEA, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD, address:104.36.82.20:4466): protocol: write tcp 159.223.143.39:49166->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614563997 (27h35m28.39549736s) [noise=true , address=104.36.82.5:4854 , piece_id=4YVTDRDA3VZ7EGFILPIP5EDO2UIHXSBPYHRB5FMN5LBRMCCNHVXQ, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy, address:104.36.82.5:4854): protocol: rpc: dial tcp 104.36.82.5:4854: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614640319 (27h35m28.395573697s) [noise=true , address=104.36.82.14:4255 , piece_id=FVMJZXGS7MPIYTUZE5FLMQ66FW54EUSUOD475CVZFGJXAGRIF65Q, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR, address:104.36.82.14:4255): protocol: write tcp 159.223.143.39:55020->104.36.82.14:4255: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614710063 (27h35m28.39564343s) [noise=true , address=104.36.82.52:4115 , piece_id=VPY4ZS2E3E3Z7X5B2TEAXBM3XPDBMTNFJ7DBZEN52BC2FQ3JIA6A, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.614958426 (27h35m28.395891796s) [noise=true , address=104.36.82.56:4248 , piece_id=QHSGEXWVL65JSHW4C2TQXWAGN4PLD7FDV2PON6KWNCMNI67ACWWA, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:1QpYBQ2PfiAwvt5EmFvjBnKgBhwnuyF6b77qJocpNe1YqSzP77, address:104.36.82.56:4248): protocol: rpc: dial tcp 104.36.82.56:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.615086152 (27h35m28.396019518s) [noise=true , address=104.36.82.50:4015 , piece_id=7LTUUCZ4DVTYMN3LTUUTPMNG4TGXQUCTXMHQFLSIN335I4VDUSDQ, seg_pos={0 0}, upload=26299]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:104.36.82.50:4015): protocol: write tcp 159.223.143.39:55852->104.36.82.50:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:27:12.615132442 (27h35m28.396065818s) [seg_pos={0 0}, upload=26299]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 01:27:12.643818824 (27h35m28.424752205s) [upload=26299]: Done uploading segments... Jul 28 01:27:12.643863116 (27h35m28.424796492s) [upload=26299]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.035441662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................X............X...............] duration: 360.108086ms FAIL (2025-07-28 01:27:08 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 360.153506ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........X....XX........X..X...........X.X....X.XX.........] duration: 395.279943ms FAIL (2025-07-28 01:26:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 395.310931ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X...X............................................X..XX.X..] duration: 394.654303ms FAIL (2025-07-28 01:26:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 8:4972 , piece_id=Y7S3Q5EZURAXHYJK4JLV6T5IWCZCMEZ73B326AGNPUV6EUS6EIEA, seg_pos={0 0}, upload=26264]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: rpc: dial tcp 104.36.82.8:4972: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Jul 28 01:24:02.780580041 (27h32m18.561513423s) [seg_pos={0 0}, upload=26264]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 01:24:02.780672601 (27h32m18.561605999s) [upload=26264]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:24:02.780678964 (27h32m18.561612327s) [upload=26264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:24:02.830372863 (27h32m18.61130625s) [upload=26264]: Done uploading segments... Jul 28 01:24:02.830396572 (27h32m18.611329958s) [upload=26264]: Upload finished. err: <nil> Jul 28 01:24:32.376837417 (27h32m48.157770784s) [upload=26272]: Starting upload Jul 28 01:24:32.376912707 (27h32m48.157846094s) [upload=26272]: Done starting upload Jul 28 01:24:32.377229908 (27h32m48.158163286s) [upload=26272]: Uploading segments... Jul 28 01:24:32.377273690 (27h32m48.158207073s) [upload=26272]: Got next segment. Inline: false Jul 28 01:24:32.377303334 (27h32m48.158236702s) [upload=26272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:24:32.377310262 (27h32m48.158243639s) [upload=26272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:24:32.406493156 (27h32m48.187426544s) [seg_pos={0 0}, upload=26272]: Begin upload segment... Jul 28 01:24:32.406630172 (27h32m48.187563559s) [seg_pos={0 0}, upload=26272]: Done begin upload segment. Jul 28 01:24:32.406678948 (27h32m48.187612334s) [noise=true , address=45.91.116.10:4083 , piece_id=ZNOMA47MEQ3APGYLX6K5ASXTGBGV4HRMFHUERUYH7OMV37FBC2WQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.406720739 (27h32m48.187654125s) [noise=true , address=77.90.31.18:4805 , piece_id=EZPRGC2M6UGTNZS2H5ZVU5ZYT6CGOFZQAIY62WUWK7CCVOPKCH6A, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.407378956 (27h32m48.188312341s) [noise=true , address=77.90.31.16:4733 , piece_id=GWDHUV5UPGOAFIMGX2VS5IWRGBKQRLBAVXXFCH7R5IDAWL24AHMQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.408056517 (27h32m48.188989903s) [noise=true , address=45.91.116.11:4119 , piece_id=5JZYKP427ZHCYETAI33EYUOSIIKGS7FOXGCGVS3WFTSGTLLQE5FQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.408963330 (27h32m48.189896719s) [noise=true , address=104.36.82.6:4876 , piece_id=FWIFZOE7JCRVKYOSO4JISR3NDQX44VYB5QL3INBEY5VZ5CXW45VQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.411970540 (27h32m48.192903917s) [noise=true , address=104.36.82.3:4765 , piece_id=UU52FP7AC7J3CDTVO2RMYLXA3ZL72Q74E4XPOFLWAOB2DW3TSFGQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.412959063 (27h32m48.193892444s) [noise=true , address=45.91.116.8:4047 , piece_id=5OPODMEAEWVDO7JVVIFKQZUBP7LZNCEZPXPRUINWSJPFEYVTPSCA, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.413496577 (27h32m48.194429967s) [noise=true , address=77.90.31.14:4661 , piece_id=O7FVERHCSZEODXSQT7FOEYMSJ3JH5JX76DLZUXVB2T7JPT6YQJHQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.414094626 (27h32m48.19502801s) [noise=true , address=77.90.31.11:4553 , piece_id=7MWOM3AYKI4T7JP7PXOGCIH54QMA5BWDITJNBBD2VOALJ6Q76AMQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.414667638 (27h32m48.195601023s) [noise=true , address=45.91.116.14:4190 , piece_id=BSG4KQZAQGAECN3ZQGMHQW2OPM7L2OKLXIKLQYWYGB3VQKHV3NIQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.415161102 (27h32m48.196094489s) [noise=true , address=104.36.82.13:4185 , piece_id=FYX5M3FMAUXW4RWEX2OLELZNNC6KU5LGXWSN3CDE7Q2LBESPCM5Q, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.421717416 (27h32m48.202650794s) [noise=true , address=77.90.31.19:4843 , piece_id=MOPV2KWI7XEX5TTAR3DC2S3T7MVTZ5HXIATUY3GNMQFJNO5AMAMQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.422404015 (27h32m48.2033374s) [noise=true , address=77.90.31.17:4772 , piece_id=INWUVWNDI756BSBZEH54FROQNK7LZD7HBZ3D4TC7REZ3TVS6JY7A, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.423067972 (27h32m48.204001356s) [noise=true , address=77.90.31.12:4592 , piece_id=ZF4DNTKCIMIVJG23LL6AMPV45GRVRT7HB4II2LQ73BYWMFNEYTRQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.423541102 (27h32m48.204474471s) [noise=true , address=45.91.116.14:4189 , piece_id=EMQHCNYSVKEBP7NXALATNHYHB4WVCMLM2CYHXPRGIRPTB7UCNUNA, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.439691259 (27h32m48.220624624s) [noise=true , address=104.36.82.9:4006 , piece_id=HUY4QAJQ5N7LUY52E7JMZ2OBMZ2NSSVM7WKBVI5DJI2CK7HZVI5Q, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.479893546 (27h32m48.260826913s) [upload=26272]: Next returned nil segment Jul 28 01:24:32.479917576 (27h32m48.260850963s) [upload=26272]: Waiting for error group managing segments... Jul 28 01:24:32.505908469 (27h32m48.286841846s) [noise=true , address=77.90.31.18:4805 , piece_id=EZPRGC2M6UGTNZS2H5ZVU5ZYT6CGOFZQAIY62WUWK7CCVOPKCH6A, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.509555144 (27h32m48.290488509s) [noise=true , address=77.90.31.14:4661 , piece_id=O7FVERHCSZEODXSQT7FOEYMSJ3JH5JX76DLZUXVB2T7JPT6YQJHQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.510148053 (27h32m48.291081421s) [noise=true , address=45.91.116.8:4047 , piece_id=5OPODMEAEWVDO7JVVIFKQZUBP7LZNCEZPXPRUINWSJPFEYVTPSCA, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.511209503 (27h32m48.292142874s) [noise=true , address=77.90.31.19:4843 , piece_id=MOPV2KWI7XEX5TTAR3DC2S3T7MVTZ5HXIATUY3GNMQFJNO5AMAMQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.520868119 (27h32m48.301801486s) [noise=true , address=77.90.31.17:4772 , piece_id=INWUVWNDI756BSBZEH54FROQNK7LZD7HBZ3D4TC7REZ3TVS6JY7A, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.521205950 (27h32m48.302139315s) [noise=true , address=45.91.116.11:4119 , piece_id=5JZYKP427ZHCYETAI33EYUOSIIKGS7FOXGCGVS3WFTSGTLLQE5FQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.521524755 (27h32m48.302458385s) [noise=true , address=77.90.31.12:4592 , piece_id=ZF4DNTKCIMIVJG23LL6AMPV45GRVRT7HB4II2LQ73BYWMFNEYTRQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.521952513 (27h32m48.302885896s) [noise=true , address=77.90.31.16:4733 , piece_id=GWDHUV5UPGOAFIMGX2VS5IWRGBKQRLBAVXXFCH7R5IDAWL24AHMQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.522191972 (27h32m48.303125336s) [noise=true , address=77.90.31.11:4553 , piece_id=7MWOM3AYKI4T7JP7PXOGCIH54QMA5BWDITJNBBD2VOALJ6Q76AMQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.522464348 (27h32m48.303397718s) [noise=true , address=45.91.116.14:4189 , piece_id=EMQHCNYSVKEBP7NXALATNHYHB4WVCMLM2CYHXPRGIRPTB7UCNUNA, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.527202663 (27h32m48.30813604s) [noise=true , address=45.91.116.10:4083 , piece_id=ZNOMA47MEQ3APGYLX6K5ASXTGBGV4HRMFHUERUYH7OMV37FBC2WQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.527864512 (27h32m48.308797885s) [noise=true , address=45.91.116.14:4190 , piece_id=BSG4KQZAQGAECN3ZQGMHQW2OPM7L2OKLXIKLQYWYGB3VQKHV3NIQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:25:02.328743539 (27h33m18.109676929s) [noise=true , address=104.36.82.13:4185 , piece_id=FYX5M3FMAUXW4RWEX2OLELZNNC6KU5LGXWSN3CDE7Q2LBESPCM5Q, seg_pos={0 0}, upload=26272]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:36610->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:02.328846292 (27h33m18.109779681s) [noise=true , address=104.36.82.9:4006 , piece_id=HUY4QAJQ5N7LUY52E7JMZ2OBMZ2NSSVM7WKBVI5DJI2CK7HZVI5Q, seg_pos={0 0}, upload=26272]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:25:02.328939350 (27h33m18.10987273s) [noise=true , address=104.36.82.3:4765 , piece_id=UU52FP7AC7J3CDTVO2RMYLXA3ZL72Q74E4XPOFLWAOB2DW3TSFGQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:46610->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:02.329043410 (27h33m18.109976781s) [noise=true , address=104.36.82.6:4876 , piece_id=FWIFZOE7JCRVKYOSO4JISR3NDQX44VYB5QL3INBEY5VZ5CXW45VQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:02.329113423 (27h33m18.110046812s) [seg_pos={0 0}, upload=26272]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:25:02.355211716 (27h33m18.1361451s) [upload=26272]: Done uploading segments... Jul 28 01:25:02.355243913 (27h33m18.136177294s) [upload=26272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.687088ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................X............X..............] duration: 225.742569ms FAIL (2025-07-28 01:26:08 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 225.784562ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........X....XX........X..X...........X.X....X.XX........] duration: 550.616047ms FAIL (2025-07-28 01:26:07 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 550.6687ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.032315835s FAIL (2025-07-28 01:25:59 UTC) ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== pload=26278]: Uploading piece... Jul 28 01:24:59.986489943 (27h33m15.767423327s) [noise=true , address=104.36.82.41:4787 , piece_id=PKTM254NYY4EDSBSINYSGYWHK2QO4UKR6JK5RKTZKUMXLDBSFUDQ, seg_pos={0 0}, upload=26278]: Uploading piece... Jul 28 01:24:59.986862945 (27h33m15.767796331s) [noise=true , address=104.36.82.55:4207 , piece_id=QNMANOW42TL52FKM23WXG4WCJXWYGS7C25HQ2XKDJ4K7RV22GKOA, seg_pos={0 0}, upload=26278]: Uploading piece... Jul 28 01:24:59.987276948 (27h33m15.768210332s) [noise=true , address=104.36.82.5:4848 , piece_id=AOGGEGNAM7A7DHW2A6ADDRJWURRCAKT2RIQD2YXKGSHSH3NTBIJQ, seg_pos={0 0}, upload=26278]: Uploading piece... Jul 28 01:24:59.987668602 (27h33m15.768601987s) [noise=true , address=104.36.82.51:4070 , piece_id=QB6CV55FDALHGSJPH5OM5ARHXQ6GTE4V7DRXA4N5U5HQPH4TYGTA, seg_pos={0 0}, upload=26278]: Uploading piece... Jul 28 01:24:59.988296919 (27h33m15.7692303s) [noise=true , address=104.36.82.46:4515 , piece_id=LTH3UQKLI47VHRKTWM7JIUU7T65E3EXLE2E7EAQFH4IIK6YNNPKA, seg_pos={0 0}, upload=26278]: Uploading piece... Jul 28 01:24:59.988652327 (27h33m15.769585712s) [noise=true , address=104.36.82.11:4136 , piece_id=3DCYX6NP3XQOU23JW2E7Z7HKCFRMH55PNA56NGBRKLDA3BTLUZ7Q, seg_pos={0 0}, upload=26278]: Uploading piece... Jul 28 01:24:59.988797690 (27h33m15.769731076s) [noise=true , address=104.36.82.10:4105 , piece_id=L23GTLLIXNOYQBK44OYSQNGUBVGMNDGVIWQK2VBLE6JCXDLI33PA, seg_pos={0 0}, upload=26278]: Uploading piece... Jul 28 01:24:59.989228169 (27h33m15.770161558s) [noise=true , address=104.36.82.53:4143 , piece_id=GWFXITMOX2TX7XGJ243HIO5X4J3RJW34UJ5NS5HDG6E5RTBEFHVA, seg_pos={0 0}, upload=26278]: Uploading piece... Jul 28 01:25:59.905382873 (27h34m15.686316258s) [upload=26278]: Next segment err: context deadline exceeded Jul 28 01:25:59.907013413 (27h34m15.687946789s) [noise=true , address=104.36.82.55:4207 , piece_id=QNMANOW42TL52FKM23WXG4WCJXWYGS7C25HQ2XKDJ4K7RV22GKOA, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: rpc: dial tcp 104.36.82.55:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907174711 (27h34m15.688108091s) [noise=true , address=104.36.82.10:4105 , piece_id=L23GTLLIXNOYQBK44OYSQNGUBVGMNDGVIWQK2VBLE6JCXDLI33PA, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen, address:104.36.82.10:4105): protocol: rpc: dial tcp 104.36.82.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907261738 (27h34m15.688195117s) [noise=true , address=104.36.82.53:4143 , piece_id=GWFXITMOX2TX7XGJ243HIO5X4J3RJW34UJ5NS5HDG6E5RTBEFHVA, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: rpc: dial tcp 104.36.82.53:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907349198 (27h34m15.688282577s) [noise=true , address=104.36.82.54:4163 , piece_id=U77NTDONSDKYANUQBPIJEL3Z3YTJRZGGYSJE7GP5DVGISCTOODQA, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: rpc: dial tcp 104.36.82.54:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907431441 (27h34m15.688364806s) [noise=true , address=104.36.82.9:4008 , piece_id=DWZB4PIRLLURPZDSVLBLXMKAXFCMXENWD4H3IS7XGMMUWOLXW5VA, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907440195 (27h34m15.688373574s) [noise=true , address=104.36.82.47:4527 , piece_id=IM3W5Q3QOGLSYIREJ2KCM3HLK4KCK4WS2APFJXOE7PUQ3CA7ZWPQ, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: rpc: dial tcp 104.36.82.47:4527: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907508058 (27h34m15.688441451s) [noise=true , address=104.36.82.44:4895 , piece_id=JGFUFQ5YW52S64OY442ROEO63SPX425STJCBJZAYM3O3RHVLRTWQ, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: rpc: dial tcp 104.36.82.44:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907526659 (27h34m15.688460032s) [noise=true , address=104.36.82.56:4243 , piece_id=OR5MO7PMWGR72RW5ZHQ3B3WOJPXRQVFKMLPYRWLGY7EAKKR6EDDA, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: rpc: dial tcp 104.36.82.56:4243: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907591129 (27h34m15.688524504s) [noise=true , address=104.36.82.42:4816 , piece_id=R5P4JBMMSP3ME2YV46OFOINEJLRCKZCMDLM27GQC6L5YZB6LFDAQ, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:1nq6pTeLCMEmtagczJkTMAdpADG5JrbSKhUjXAHy9oeS1pTpSx, address:104.36.82.42:4816): protocol: rpc: dial tcp 104.36.82.42:4816: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907610262 (27h34m15.688543639s) [noise=true , address=104.36.82.52:4113 , piece_id=CX45WMUCECCQX7X7XM3WJYXLC442CHZOLE4KRQ7VYCPFW7G2C3PQ, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou, address:104.36.82.52:4113): protocol: write tcp 159.223.143.39:36698->104.36.82.52:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907673283 (27h34m15.688606644s) [noise=true , address=104.36.82.41:4787 , piece_id=PKTM254NYY4EDSBSINYSGYWHK2QO4UKR6JK5RKTZKUMXLDBSFUDQ, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:12aKqg1aNsv8Yi7fE4asGz8bijcPewWDvUukKcf5SFnzgARju43, address:104.36.82.41:4787): protocol: write tcp 159.223.143.39:34862->104.36.82.41:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907686715 (27h34m15.688620091s) [noise=true , address=104.36.82.54:4171 , piece_id=IDRNRHNIYZB3UG36SZZ5J4K5L7R7STWHK3COZ5EVBA7DGMCZEBJQ, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: rpc: dial tcp 104.36.82.54:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907735092 (27h34m15.688668459s) [noise=true , address=104.36.82.51:4070 , piece_id=QB6CV55FDALHGSJPH5OM5ARHXQ6GTE4V7DRXA4N5U5HQPH4TYGTA, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me, address:104.36.82.51:4070): protocol: rpc: dial tcp 104.36.82.51:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907818575 (27h34m15.688751953s) [noise=true , address=104.36.82.22:4518 , piece_id=LXHPDJJQQKWZCN7JBO2CAVNTL5OKQWA5SGCASCJNQ2LGYHPZNAXQ, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:1ozLDHDa9ankBimUBjpM91VrraKMptSCQPoDS9yWu2tjVErasU, address:104.36.82.22:4518): protocol: rpc: dial tcp 104.36.82.22:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907830095 (27h34m15.688763461s) [noise=true , address=104.36.82.46:4515 , piece_id=LTH3UQKLI47VHRKTWM7JIUU7T65E3EXLE2E7EAQFH4IIK6YNNPKA, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: rpc: dial tcp 104.36.82.46:4515: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907903396 (27h34m15.688836773s) [noise=true , address=104.36.82.5:4848 , piece_id=AOGGEGNAM7A7DHW2A6ADDRJWURRCAKT2RIQD2YXKGSHSH3NTBIJQ, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:1xvod7N8eEwunLDYWAvVUP1Cp1CdEr3quJUMG5kWyBrwzzr9vp, address:104.36.82.5:4848): protocol: rpc: dial tcp 104.36.82.5:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.907935651 (27h34m15.688869032s) [noise=true , address=104.36.82.46:4512 , piece_id=S62BBP35IO67KS6G236I5ZSUAGYK2QH2KYLNDCJ3E5WCNYZW4K4A, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: rpc: dial tcp 104.36.82.46:4512: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.908012417 (27h34m15.688945796s) [noise=true , address=104.36.82.11:4136 , piece_id=3DCYX6NP3XQOU23JW2E7Z7HKCFRMH55PNA56NGBRKLDA3BTLUZ7Q, seg_pos={0 0}, upload=26278]: Done uploading piece. err: ecclient: upload failed (node:127n8CMbZQ7haiv6tGHrVg9V8Yf12sYkcbL9mEbSpQ4WkT9isnp, address:104.36.82.11:4136): protocol: write tcp 159.223.143.39:58292->104.36.82.11:4136: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:59.908260363 (27h34m15.689193745s) [seg_pos={0 0}, upload=26278]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.032330531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................X............X.............] duration: 216.386535ms FAIL (2025-07-28 01:25:51 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 216.417172ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [............X....XX........X..X...........X.X....X.XX.......] duration: 474.344793ms FAIL (2025-07-28 01:25:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.439864ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...X............................................X..XX.X.] duration: 1.080044768s FAIL (2025-07-28 01:25:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 8:4972 , piece_id=Y7S3Q5EZURAXHYJK4JLV6T5IWCZCMEZ73B326AGNPUV6EUS6EIEA, seg_pos={0 0}, upload=26264]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: rpc: dial tcp 104.36.82.8:4972: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Jul 28 01:24:02.780580041 (27h32m18.561513423s) [seg_pos={0 0}, upload=26264]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 01:24:02.780672601 (27h32m18.561605999s) [upload=26264]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:24:02.780678964 (27h32m18.561612327s) [upload=26264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:24:02.830372863 (27h32m18.61130625s) [upload=26264]: Done uploading segments... Jul 28 01:24:02.830396572 (27h32m18.611329958s) [upload=26264]: Upload finished. err: <nil> Jul 28 01:24:32.376837417 (27h32m48.157770784s) [upload=26272]: Starting upload Jul 28 01:24:32.376912707 (27h32m48.157846094s) [upload=26272]: Done starting upload Jul 28 01:24:32.377229908 (27h32m48.158163286s) [upload=26272]: Uploading segments... Jul 28 01:24:32.377273690 (27h32m48.158207073s) [upload=26272]: Got next segment. Inline: false Jul 28 01:24:32.377303334 (27h32m48.158236702s) [upload=26272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:24:32.377310262 (27h32m48.158243639s) [upload=26272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:24:32.406493156 (27h32m48.187426544s) [seg_pos={0 0}, upload=26272]: Begin upload segment... Jul 28 01:24:32.406630172 (27h32m48.187563559s) [seg_pos={0 0}, upload=26272]: Done begin upload segment. Jul 28 01:24:32.406678948 (27h32m48.187612334s) [noise=true , address=45.91.116.10:4083 , piece_id=ZNOMA47MEQ3APGYLX6K5ASXTGBGV4HRMFHUERUYH7OMV37FBC2WQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.406720739 (27h32m48.187654125s) [noise=true , address=77.90.31.18:4805 , piece_id=EZPRGC2M6UGTNZS2H5ZVU5ZYT6CGOFZQAIY62WUWK7CCVOPKCH6A, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.407378956 (27h32m48.188312341s) [noise=true , address=77.90.31.16:4733 , piece_id=GWDHUV5UPGOAFIMGX2VS5IWRGBKQRLBAVXXFCH7R5IDAWL24AHMQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.408056517 (27h32m48.188989903s) [noise=true , address=45.91.116.11:4119 , piece_id=5JZYKP427ZHCYETAI33EYUOSIIKGS7FOXGCGVS3WFTSGTLLQE5FQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.408963330 (27h32m48.189896719s) [noise=true , address=104.36.82.6:4876 , piece_id=FWIFZOE7JCRVKYOSO4JISR3NDQX44VYB5QL3INBEY5VZ5CXW45VQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.411970540 (27h32m48.192903917s) [noise=true , address=104.36.82.3:4765 , piece_id=UU52FP7AC7J3CDTVO2RMYLXA3ZL72Q74E4XPOFLWAOB2DW3TSFGQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.412959063 (27h32m48.193892444s) [noise=true , address=45.91.116.8:4047 , piece_id=5OPODMEAEWVDO7JVVIFKQZUBP7LZNCEZPXPRUINWSJPFEYVTPSCA, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.413496577 (27h32m48.194429967s) [noise=true , address=77.90.31.14:4661 , piece_id=O7FVERHCSZEODXSQT7FOEYMSJ3JH5JX76DLZUXVB2T7JPT6YQJHQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.414094626 (27h32m48.19502801s) [noise=true , address=77.90.31.11:4553 , piece_id=7MWOM3AYKI4T7JP7PXOGCIH54QMA5BWDITJNBBD2VOALJ6Q76AMQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.414667638 (27h32m48.195601023s) [noise=true , address=45.91.116.14:4190 , piece_id=BSG4KQZAQGAECN3ZQGMHQW2OPM7L2OKLXIKLQYWYGB3VQKHV3NIQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.415161102 (27h32m48.196094489s) [noise=true , address=104.36.82.13:4185 , piece_id=FYX5M3FMAUXW4RWEX2OLELZNNC6KU5LGXWSN3CDE7Q2LBESPCM5Q, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.421717416 (27h32m48.202650794s) [noise=true , address=77.90.31.19:4843 , piece_id=MOPV2KWI7XEX5TTAR3DC2S3T7MVTZ5HXIATUY3GNMQFJNO5AMAMQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.422404015 (27h32m48.2033374s) [noise=true , address=77.90.31.17:4772 , piece_id=INWUVWNDI756BSBZEH54FROQNK7LZD7HBZ3D4TC7REZ3TVS6JY7A, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.423067972 (27h32m48.204001356s) [noise=true , address=77.90.31.12:4592 , piece_id=ZF4DNTKCIMIVJG23LL6AMPV45GRVRT7HB4II2LQ73BYWMFNEYTRQ, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.423541102 (27h32m48.204474471s) [noise=true , address=45.91.116.14:4189 , piece_id=EMQHCNYSVKEBP7NXALATNHYHB4WVCMLM2CYHXPRGIRPTB7UCNUNA, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.439691259 (27h32m48.220624624s) [noise=true , address=104.36.82.9:4006 , piece_id=HUY4QAJQ5N7LUY52E7JMZ2OBMZ2NSSVM7WKBVI5DJI2CK7HZVI5Q, seg_pos={0 0}, upload=26272]: Uploading piece... Jul 28 01:24:32.479893546 (27h32m48.260826913s) [upload=26272]: Next returned nil segment Jul 28 01:24:32.479917576 (27h32m48.260850963s) [upload=26272]: Waiting for error group managing segments... Jul 28 01:24:32.505908469 (27h32m48.286841846s) [noise=true , address=77.90.31.18:4805 , piece_id=EZPRGC2M6UGTNZS2H5ZVU5ZYT6CGOFZQAIY62WUWK7CCVOPKCH6A, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.509555144 (27h32m48.290488509s) [noise=true , address=77.90.31.14:4661 , piece_id=O7FVERHCSZEODXSQT7FOEYMSJ3JH5JX76DLZUXVB2T7JPT6YQJHQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.510148053 (27h32m48.291081421s) [noise=true , address=45.91.116.8:4047 , piece_id=5OPODMEAEWVDO7JVVIFKQZUBP7LZNCEZPXPRUINWSJPFEYVTPSCA, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.511209503 (27h32m48.292142874s) [noise=true , address=77.90.31.19:4843 , piece_id=MOPV2KWI7XEX5TTAR3DC2S3T7MVTZ5HXIATUY3GNMQFJNO5AMAMQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.520868119 (27h32m48.301801486s) [noise=true , address=77.90.31.17:4772 , piece_id=INWUVWNDI756BSBZEH54FROQNK7LZD7HBZ3D4TC7REZ3TVS6JY7A, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.521205950 (27h32m48.302139315s) [noise=true , address=45.91.116.11:4119 , piece_id=5JZYKP427ZHCYETAI33EYUOSIIKGS7FOXGCGVS3WFTSGTLLQE5FQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.521524755 (27h32m48.302458385s) [noise=true , address=77.90.31.12:4592 , piece_id=ZF4DNTKCIMIVJG23LL6AMPV45GRVRT7HB4II2LQ73BYWMFNEYTRQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.521952513 (27h32m48.302885896s) [noise=true , address=77.90.31.16:4733 , piece_id=GWDHUV5UPGOAFIMGX2VS5IWRGBKQRLBAVXXFCH7R5IDAWL24AHMQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.522191972 (27h32m48.303125336s) [noise=true , address=77.90.31.11:4553 , piece_id=7MWOM3AYKI4T7JP7PXOGCIH54QMA5BWDITJNBBD2VOALJ6Q76AMQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.522464348 (27h32m48.303397718s) [noise=true , address=45.91.116.14:4189 , piece_id=EMQHCNYSVKEBP7NXALATNHYHB4WVCMLM2CYHXPRGIRPTB7UCNUNA, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.527202663 (27h32m48.30813604s) [noise=true , address=45.91.116.10:4083 , piece_id=ZNOMA47MEQ3APGYLX6K5ASXTGBGV4HRMFHUERUYH7OMV37FBC2WQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:24:32.527864512 (27h32m48.308797885s) [noise=true , address=45.91.116.14:4190 , piece_id=BSG4KQZAQGAECN3ZQGMHQW2OPM7L2OKLXIKLQYWYGB3VQKHV3NIQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: <nil> Jul 28 01:25:02.328743539 (27h33m18.109676929s) [noise=true , address=104.36.82.13:4185 , piece_id=FYX5M3FMAUXW4RWEX2OLELZNNC6KU5LGXWSN3CDE7Q2LBESPCM5Q, seg_pos={0 0}, upload=26272]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:36610->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:02.328846292 (27h33m18.109779681s) [noise=true , address=104.36.82.9:4006 , piece_id=HUY4QAJQ5N7LUY52E7JMZ2OBMZ2NSSVM7WKBVI5DJI2CK7HZVI5Q, seg_pos={0 0}, upload=26272]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:25:02.328939350 (27h33m18.10987273s) [noise=true , address=104.36.82.3:4765 , piece_id=UU52FP7AC7J3CDTVO2RMYLXA3ZL72Q74E4XPOFLWAOB2DW3TSFGQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:46610->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:02.329043410 (27h33m18.109976781s) [noise=true , address=104.36.82.6:4876 , piece_id=FWIFZOE7JCRVKYOSO4JISR3NDQX44VYB5QL3INBEY5VZ5CXW45VQ, seg_pos={0 0}, upload=26272]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:25:02.329113423 (27h33m18.110046812s) [seg_pos={0 0}, upload=26272]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:25:02.355211716 (27h33m18.1361451s) [upload=26272]: Done uploading segments... Jul 28 01:25:02.355243913 (27h33m18.136177294s) [upload=26272]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.080082082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................X............X............] duration: 281.036ms FAIL (2025-07-28 01:25:17 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 281.066721ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............X....XX........X..X...........X.X....X.XX......] duration: 507.919858ms FAIL (2025-07-28 01:25:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 507.952489ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)