Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration:
358.524887ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:36:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 358.564882ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
391.445834ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:35:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 391.491514ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X.X...] duration:
368.537013ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:35:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 368.564475ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................................XX..X.X....] duration:
458.80374ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:35:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 458.824032ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X..X...] duration:
506.045039ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:35:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 506.072171ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X.....X...] duration:
334.218959ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:35:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 334.24397ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..................................................X...XX.X..] duration:
592.062481ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:35:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 592.090584ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X..X..] duration:
508.818508ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:35:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 508.897134ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................................................XX..X.X...] duration:
463.359956ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:35:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 463.386031ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...................................................X...XX.X.] duration:
420.950295ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:34:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 420.98835ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
358.664129ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:34:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 358.692023ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration:
361.39318ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:34:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 361.419523ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................................................XX..X.X..] duration:
443.405995ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:34:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 443.429398ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X.....X..] duration:
354.686373ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:34:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 354.712469ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration:
510.887709ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:34:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 510.906311ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
345.864822ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:34:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 345.889248ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................X.X.] duration:
390.275938ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:34:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 390.295395ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X.....X.] duration:
359.870437ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:33:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 359.889912ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................................................XX..X.X.] duration:
7.750278229s trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:33:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.750301847s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....................................................X...XX.X] duration:
5.00055944s trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:33:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000582427s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................................................XX..X.X] duration:
30.001033133s trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:33:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001056256s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
327.12347ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:33:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 327.187918ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
363.686294ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:32:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 363.710369ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.....................................................X...XX.] duration:
412.555145ms trace:
https://jaeger.dev.storj.io/trace/9fbd8eb29b1fce
FAIL (2025-11-22 12:32:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 412.578018ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
538.013721ms trace:
https://jaeger.dev.storj.io/trace/7a4bcbc76442759
FAIL (2025-11-22 12:32:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 538.03214ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................................................XX..X.] duration:
497.415159ms trace:
https://jaeger.dev.storj.io/trace/268bcafc480c1964
FAIL (2025-11-22 12:32:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 497.451883ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
372.973637ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:31:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 372.998892ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
432.850967ms trace:
https://jaeger.dev.storj.io/trace/3ad0c5063740d2ff
FAIL (2025-11-22 12:31:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
Nov 22 12:30:47.245783177 (266h44m12.013466603s) [upload=277495]: Starting upload
Nov 22 12:30:47.245837282 (266h44m12.013520714s) [upload=277495]: Done starting upload
Nov 22 12:30:47.246118756 (266h44m12.013802169s) [upload=277495]: Uploading segments...
Nov 22 12:30:47.246164769 (266h44m12.013848198s) [upload=277495]: Got next segment. Inline: false
Nov 22 12:30:47.246201768 (266h44m12.013885179s) [upload=277495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:47.246205660 (266h44m12.013889086s) [upload=277495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:47.272577767 (266h44m12.040261203s) [seg_pos={0 0}, upload=277495]: Begin upload segment...
Nov 22 12:30:47.272637564 (266h44m12.040320981s) [seg_pos={0 0}, upload=277495]: Done begin upload segment.
Nov 22 12:30:47.272666732 (266h44m12.040350157s) [noise=true , address=77.90.31.12:4600 , piece_id=DTEZIYZ5PNICLTJPWJNXPYFP5VHFKIQLV5XB5RILGPVFKYUVHW3A, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.272763351 (266h44m12.040446785s) [noise=true , address=77.90.31.19:4857 , piece_id=W5CSC7J6QOER7I2T7PZSEOZZMYQGMYOXMDGIL3QWLUQDJ2NJNWZA, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.273380146 (266h44m12.041063581s) [noise=true , address=145.79.227.12:4786 , piece_id=HSEOVBW5GSNYRRIBXJEYIJ4B3QEHEOR54F27YYBI7YY7SBXLQQ4A, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.273927088 (266h44m12.041610526s) [noise=true , address=77.90.31.11:4587 , piece_id=TOUCPWIV6I7B6KWB7BG6NML55ZB4ZEXTDZJC6CYHZNQ3DKRJ63EA, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.274540376 (266h44m12.042223812s) [noise=true , address=145.79.227.15:4897 , piece_id=2YE67U6JK5OYQYXKBY67MPMNNTKHJARSDYWVDZQLFSMDMHYIH2NQ, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.275049678 (266h44m12.042733105s) [noise=true , address=45.91.116.10:4098 , piece_id=DL2MJ6LVVG2TBVHB6POE7TOPMWZDQNYCJUGOFIR7LV2PVEL6E6HA, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.275579988 (266h44m12.043263414s) [noise=true , address=77.90.31.10:4545 , piece_id=W5CYEW7TNRAFOUPFL7HI2CTIDSRML3AW4NDN7JT4FWJXYMIGXFRA, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.275856000 (266h44m12.043539429s) [noise=true , address=45.91.116.8:4065 , piece_id=O7PTIRRPMCSBBJH6HWVZYLSB5LZQPJ6RWCOOZVETYEKTUSTWW2MQ, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.276362296 (266h44m12.044045727s) [noise=true , address=145.79.227.49:4351 , piece_id=7SEOVSVOCKZ56T4YTMATTORWUXHKLGR5IVM5ZAYC6BFPGJI5X6NQ, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.276768629 (266h44m12.044452058s) [noise=true , address=45.91.116.12:4235 , piece_id=VN4TGP3HEOJZBISCD4AUN6Q3TYEF4MNVKRQVORJWDNSGAXOOJ5SQ, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.277331057 (266h44m12.045014489s) [noise=true , address=145.79.227.32:4844 , piece_id=IN6VHYJOOA5RZSL3R3I46NPOGNK73CBVVILSCHFTTK2ONGEPQ4EQ, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.277685572 (266h44m12.045369004s) [noise=true , address=145.79.227.52:4462 , piece_id=BDMVRN5FUTZBGWQ52PNRULGAACUBAMUPMC6LEBOQY7XJBS25CAWQ, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.278310272 (266h44m12.0459937s) [noise=true , address=77.90.31.14:4696 , piece_id=LTU3ABFK4XINYBXGDDRWRCKCJRO7GGGPWLO5RGPGYPFOGEU7VG2Q, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.278712244 (266h44m12.046395672s) [noise=true , address=45.91.116.6:4994 , piece_id=JLTFPCHDAAYSGLSSO7HHEIRDVY4XPKHVEWTKOYU4AGOO6FZ6QVIQ, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.279095380 (266h44m12.046778835s) [noise=true , address=45.91.116.5:4322 , piece_id=3Q7IZAWPI6SX42ALSJPEGJYQIGIGWX63MH5J2SF4KBZ6JSDNQVCQ, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.279392209 (266h44m12.04707564s) [noise=true , address=45.91.116.13:4180 , piece_id=VMOATGKOOG5ZZLD7WWWSAUFPWG7ACK3FJ4U7FFJRDS6TTPTMROOA, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.279765487 (266h44m12.047448922s) [noise=true , address=77.90.31.15:4732 , piece_id=UVGC7LHZGDK4ENXIWI2IVEVKT7VG2FRXQGN6NY3CAJH7DR4YOYZQ, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.280897556 (266h44m12.048580984s) [noise=true , address=77.90.31.18:4826 , piece_id=IGLWWJMB5OEQ3XDWH6VHAXBMRL75UOXAVZ6WX4EG2MZTYUUUANVQ, seg_pos={0 0}, upload=277495]: Uploading piece...
Nov 22 12:30:47.344136812 (266h44m12.111820241s) [upload=277495]: Next returned nil segment
Nov 22 12:30:47.344154586 (266h44m12.111838017s) [upload=277495]: Waiting for error group managing segments...
Nov 22 12:30:47.351861657 (266h44m12.119545082s) [noise=true , address=145.79.227.12:4786 , piece_id=HSEOVBW5GSNYRRIBXJEYIJ4B3QEHEOR54F27YYBI7YY7SBXLQQ4A, seg_pos={0 0}, upload=277495]: Done uploading piece. err: <nil>
Nov 22 12:30:47.351901957 (266h44m12.119585385s) [noise=true , address=145.79.227.52:4462 , piece_id=BDMVRN5FUTZBGWQ52PNRULGAACUBAMUPMC6LEBOQY7XJBS25CAWQ, seg_pos={0 0}, upload=277495]: Done uploading piece. err: <nil>
Nov 22 12:30:47.352518300 (266h44m12.120201725s) [noise=true , address=145.79.227.32:4844 , piece_id=IN6VHYJOOA5RZSL3R3I46NPOGNK73CBVVILSCHFTTK2ONGEPQ4EQ, seg_pos={0 0}, upload=277495]: Done uploading piece. err: <nil>
Nov 22 12:30:47.352593135 (266h44m12.120276563s) [noise=true , address=145.79.227.15:4897 , piece_id=2YE67U6JK5OYQYXKBY67MPMNNTKHJARSDYWVDZQLFSMDMHYIH2NQ, seg_pos={0 0}, upload=277495]: Done uploading piece. err: <nil>
Nov 22 12:30:47.353018671 (266h44m12.120702096s) [noise=true , address=145.79.227.49:4351 , piece_id=7SEOVSVOCKZ56T4YTMATTORWUXHKLGR5IVM5ZAYC6BFPGJI5X6NQ, seg_pos={0 0}, upload=277495]: Done uploading piece. err: <nil>
Nov 22 12:31:36.794828057 (266h45m1.562511482s) [noise=true , address=45.91.116.12:4235 , piece_id=VN4TGP3HEOJZBISCD4AUN6Q3TYEF4MNVKRQVORJWDNSGAXOOJ5SQ, seg_pos={0 0}, upload=277495]: Done uploading piece. err: <nil>
Nov 22 12:31:47.209124704 (266h45m11.976808134s) [noise=true , address=77.90.31.12:4600 , piece_id=DTEZIYZ5PNICLTJPWJNXPYFP5VHFKIQLV5XB5RILGPVFKYUVHW3A, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: write tcp 159.223.143.39:38472->77.90.31.12:4600: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.209214509 (266h45m11.976897937s) [noise=true , address=45.91.116.10:4098 , piece_id=DL2MJ6LVVG2TBVHB6POE7TOPMWZDQNYCJUGOFIR7LV2PVEL6E6HA, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: write tcp 159.223.143.39:55696->45.91.116.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.209380917 (266h45m11.977064346s) [noise=true , address=77.90.31.10:4545 , piece_id=W5CYEW7TNRAFOUPFL7HI2CTIDSRML3AW4NDN7JT4FWJXYMIGXFRA, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: write tcp 159.223.143.39:47180->77.90.31.10:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.209401268 (266h45m11.977084694s) [noise=true , address=77.90.31.19:4857 , piece_id=W5CSC7J6QOER7I2T7PZSEOZZMYQGMYOXMDGIL3QWLUQDJ2NJNWZA, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, address:77.90.31.19:4857): protocol: read tcp 159.223.143.39:52026->77.90.31.19:4857: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.209569289 (266h45m11.977252719s) [noise=true , address=77.90.31.14:4696 , piece_id=LTU3ABFK4XINYBXGDDRWRCKCJRO7GGGPWLO5RGPGYPFOGEU7VG2Q, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:52870->77.90.31.14:4696: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.209608523 (266h45m11.97729195s) [noise=true , address=45.91.116.5:4322 , piece_id=3Q7IZAWPI6SX42ALSJPEGJYQIGIGWX63MH5J2SF4KBZ6JSDNQVCQ, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:12W3qDq8BssS8V6osW9NwFHZHMJcCRfjYP8qKwphE5SenVewWXy, address:45.91.116.5:4322): protocol: write tcp 159.223.143.39:37390->45.91.116.5:4322: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.209918861 (266h45m11.977602293s) [noise=true , address=45.91.116.13:4180 , piece_id=VMOATGKOOG5ZZLD7WWWSAUFPWG7ACK3FJ4U7FFJRDS6TTPTMROOA, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: write tcp 159.223.143.39:34128->45.91.116.13:4180: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.209955419 (266h45m11.977638844s) [noise=true , address=77.90.31.15:4732 , piece_id=UVGC7LHZGDK4ENXIWI2IVEVKT7VG2FRXQGN6NY3CAJH7DR4YOYZQ, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: write tcp 159.223.143.39:57400->77.90.31.15:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.210164548 (266h45m11.977847981s) [noise=true , address=77.90.31.18:4826 , piece_id=IGLWWJMB5OEQ3XDWH6VHAXBMRL75UOXAVZ6WX4EG2MZTYUUUANVQ, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ, address:77.90.31.18:4826): protocol: write tcp 159.223.143.39:35786->77.90.31.18:4826: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.210173108 (266h45m11.977856556s) [noise=true , address=45.91.116.8:4065 , piece_id=O7PTIRRPMCSBBJH6HWVZYLSB5LZQPJ6RWCOOZVETYEKTUSTWW2MQ, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:17EPCQmYb816aegE9vPSoJYByMUEvhP4DUiabis2gh9QjXQC3s, address:45.91.116.8:4065): protocol: write tcp 159.223.143.39:45786->45.91.116.8:4065: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.210412813 (266h45m11.978096239s) [noise=true , address=77.90.31.11:4587 , piece_id=TOUCPWIV6I7B6KWB7BG6NML55ZB4ZEXTDZJC6CYHZNQ3DKRJ63EA, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: write tcp 159.223.143.39:46536->77.90.31.11:4587: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.210734509 (266h45m11.978417934s) [noise=true , address=45.91.116.6:4994 , piece_id=JLTFPCHDAAYSGLSSO7HHEIRDVY4XPKHVEWTKOYU4AGOO6FZ6QVIQ, seg_pos={0 0}, upload=277495]: Done uploading piece. err: ecclient: upload failed (node:13bDFHbqDxbPr7n4PTQYCXE151sUUFPkidaY1KBZSPcPBBAnFo, address:45.91.116.6:4994): protocol: read tcp 159.223.143.39:34774->45.91.116.6:4994: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.210772469 (266h45m11.978455902s) [seg_pos={0 0}, upload=277495]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:31:47.228805818 (266h45m11.996489248s) [upload=277495]: Done uploading segments...
Nov 22 12:31:47.228857285 (266h45m11.99654072s) [upload=277495]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 432.868631ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
508.404582ms trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:31:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 508.43472ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
542.189652ms trace:
https://jaeger.dev.storj.io/trace/7a4bcbc76442759
FAIL (2025-11-22 12:31:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 542.219708ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.00076234s trace:
https://jaeger.dev.storj.io/trace/7a4bcbc76442759
FAIL (2025-11-22 12:31:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000772581s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
15.772918111s trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:31:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 15.772945117s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................................................XX..X] duration:
30.001011314s trace:
https://jaeger.dev.storj.io/trace/268bcafc480c1964
FAIL (2025-11-22 12:31:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001022328s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
15.588749252s trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:31:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 15.588803013s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [......................................................X...XX] duration:
5.001362477s trace:
https://jaeger.dev.storj.io/trace/9fbd8eb29b1fce
FAIL (2025-11-22 12:31:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001393614s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
370.741604ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:31:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 370.769778ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................................................X...X] duration:
5.001432379s trace:
https://jaeger.dev.storj.io/trace/274046331542fe3e
FAIL (2025-11-22 12:30:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00145873s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
456.789613ms trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:30:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 456.816404ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................................................X...] duration:
396.699158ms trace:
https://jaeger.dev.storj.io/trace/76878535c6f212a7
FAIL (2025-11-22 12:30:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 396.747278ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
476.331984ms trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:30:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 476.365186ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
460.652063ms trace:
https://jaeger.dev.storj.io/trace/7e867fdf9a6b34ed
FAIL (2025-11-22 12:30:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 460.677467ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
384.903674ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:29:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 384.932859ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..] duration:
413.240288ms trace:
https://jaeger.dev.storj.io/trace/76878535c6f212a7
FAIL (2025-11-22 12:29:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 413.596863ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
449.385618ms trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:29:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 449.410173ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.028569743s trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:29:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 1m0.028598706s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.] duration:
385.531471ms trace:
https://jaeger.dev.storj.io/trace/76878535c6f212a7
FAIL (2025-11-22 12:29:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 385.563639ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
478.216442ms trace:
https://jaeger.dev.storj.io/trace/7e867fdf9a6b34ed
FAIL (2025-11-22 12:28:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 478.242297ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.002587843s trace:
https://jaeger.dev.storj.io/trace/7e867fdf9a6b34ed
FAIL (2025-11-22 12:28:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00260934s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
1.746096247s trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:28:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 1.746120561s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.033574738s trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:28:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.033596282s
core
uplink
us-select-1
|