Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 411.376774ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:11:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 411.401956ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 486.037849ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:11:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 486.061104ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 466.271797ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:11:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 466.29983ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........XX..................................................] duration: 730.591257ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:11:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 730.64476ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 324.619965ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:11:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 324.645562ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 541.014133ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:10:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 541.030323ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 505.714083ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:10:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 505.740652ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 520.152493ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:10:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 520.183061ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........XX.................................................] duration: 419.351781ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:10:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 419.372738ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........XX................................................] duration: 763.912547ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:10:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 763.941536ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 537.721796ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:10:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 537.751934ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 536.117631ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:09:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 536.134397ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 339.701619ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:09:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 339.725966ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 525.714807ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:09:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 525.738915ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 889.215693ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:09:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 889.239866ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 469.888166ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:09:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 469.916063ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 488.787378ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:09:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 488.810947ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........XX...............................................] duration: 454.415045ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:08:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 454.444848ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 364.345296ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:08:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 364.37065ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 473.972579ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:08:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 473.995775ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............XX..............................................] duration: 461.929685ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:08:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.954071ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 440.057382ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:08:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 440.086335ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 341.045027ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:08:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 341.074434ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 518.435425ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:07:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 518.463848ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 350.247309ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:07:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 350.277661ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 625.662445ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:07:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 625.679044ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............XX.............................................] duration: 476.874457ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:07:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 476.902648ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 547.561428ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:07:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 547.578955ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 491.304775ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:06:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 491.331809ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 395.081006ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:06:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 395.105543ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............XX............................................] duration: 653.441885ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:06:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 653.469843ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 601.062419ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:06:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 601.095017ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 493.729366ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:06:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 493.784185ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 402.337099ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:05:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 402.378793ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 557.852052ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:05:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 557.88742ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............XX...........................................] duration: 449.68497ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:05:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 449.708849ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 637.731743ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:05:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 637.914427ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 573.198501ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:04:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 573.223271ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................XX..........................................] duration: 456.436486ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:04:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.513808ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 393.018426ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:04:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 393.235605ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 530.802312ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:04:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 530.831982ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................XX.........................................] duration: 434.042237ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:03:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 434.064542ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 467.022284ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:03:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 467.060807ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 537.489236ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:03:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 537.552607ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 386.26973ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:03:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 386.293618ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 492.246214ms trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 23:03:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 492.271884ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 504.064487ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:03:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 504.087589ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................XX........................................] duration: 457.901952ms trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 23:03:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.931604ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 369.792254ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 23:02:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 369.867544ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 505.710329ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 23:02:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 505.759381ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)